Nov 03 06:56:18.018446 ip-10-0-136-184 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 06:56:18.018458 ip-10-0-136-184 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 06:56:18.018468 ip-10-0-136-184 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 06:56:18.018901 ip-10-0-136-184 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 06:56:28.024818 ip-10-0-136-184 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 06:56:28.024830 ip-10-0-136-184 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e8f9580c58ab4870b9422b040345e9d7 -- Nov 03 06:58:18.191278 ip-10-0-136-184 systemd[1]: Starting Kubernetes Kubelet... Nov 03 06:58:18.836139 ip-10-0-136-184 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:58:18.836139 ip-10-0-136-184 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 06:58:18.836139 ip-10-0-136-184 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:58:18.836139 ip-10-0-136-184 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:58:18.840451 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840330 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.840451 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840451 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.840451 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840455 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840458 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840461 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840464 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840472 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840475 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840478 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840481 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840484 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840486 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840489 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840491 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840494 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840496 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840499 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840503 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840506 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840510 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840514 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.840547 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840516 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840519 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840522 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840524 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840527 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840529 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840532 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840534 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840537 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840539 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840542 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840544 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840546 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840549 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840552 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840554 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840557 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840560 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840563 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840566 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.840973 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840568 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840571 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840573 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840575 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840578 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840581 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840583 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840585 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840588 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840590 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840593 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840595 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840597 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840600 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840603 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840606 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840608 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840611 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840613 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840616 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.841446 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840618 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840620 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840623 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840625 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840630 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840635 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840638 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840641 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840643 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840646 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840649 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840652 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840655 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840658 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840660 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840663 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840665 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840668 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840670 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.841914 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840673 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840675 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840678 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840680 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840684 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840686 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840689 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840692 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840694 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840697 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840699 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840702 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840704 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840707 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840709 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840712 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840714 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840717 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840719 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840721 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.842365 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840724 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840727 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840730 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840733 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840735 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.840738 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841529 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841536 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841539 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841542 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841545 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841547 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841550 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841553 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841556 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841558 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841561 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841564 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841567 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.842822 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841569 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841572 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841575 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841577 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841580 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841582 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841584 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841587 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841590 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841592 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841595 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841597 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841599 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841602 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841604 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841606 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841609 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841611 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841614 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841617 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.843277 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841620 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841622 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841624 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841627 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841629 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841632 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841634 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841637 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841639 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841642 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841645 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841647 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841650 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841652 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841655 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841657 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841659 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841662 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841665 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.843732 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841667 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841670 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841672 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841675 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841678 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841680 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841682 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841685 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841687 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841689 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841693 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841696 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841698 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841701 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841705 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841710 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841714 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841716 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841719 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.844253 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841722 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841724 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841726 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841729 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841732 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841734 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841737 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841740 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841742 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841745 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841747 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841750 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841752 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841755 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841757 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841760 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841762 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841764 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841767 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841769 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.844815 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841771 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841774 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841776 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841781 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841784 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841786 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841789 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841792 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841794 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841797 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841800 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841802 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841805 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841808 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.841811 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842286 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842291 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842293 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842296 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842299 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.845312 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842302 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842304 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842307 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842309 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842312 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842315 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842317 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842320 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842322 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842325 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842327 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842330 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842332 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842335 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842337 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842340 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842343 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842345 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842349 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.845824 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842354 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842358 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842361 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842364 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842367 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842370 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842372 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842375 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842378 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842380 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842383 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842386 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842388 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842391 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842393 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842396 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842399 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842401 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842404 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.846413 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842406 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842409 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842411 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842413 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842416 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842418 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842421 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842423 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842425 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842428 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842430 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842433 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842436 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842439 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842442 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842444 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842447 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842449 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842452 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.846876 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842454 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842457 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842459 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842463 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842466 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842468 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842471 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842473 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842476 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842478 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842481 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842483 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842485 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842488 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842490 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842493 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842495 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842498 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842500 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842503 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.847329 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842505 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842508 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842510 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842513 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842515 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842517 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842520 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842522 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842525 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842527 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842530 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842533 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842535 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842538 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842541 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842543 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842546 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842548 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842551 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842553 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.847799 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842556 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842558 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842560 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.842563 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843527 2572 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843537 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843542 2572 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843547 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843551 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843555 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843559 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843563 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843567 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843570 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843573 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843577 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843580 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843582 2572 flags.go:64] FLAG: --cgroup-root="" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843585 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843588 2572 flags.go:64] FLAG: --client-ca-file="" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843591 2572 flags.go:64] FLAG: --cloud-provider="external" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843595 2572 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843599 2572 flags.go:64] FLAG: --cluster-domain="" Nov 03 06:58:18.848294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843602 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843605 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843608 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843612 2572 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843615 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843618 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843621 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843629 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843634 2572 flags.go:64] FLAG: --contention-profiling="false" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843637 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843640 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843643 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843646 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843651 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843654 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843657 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843660 2572 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843663 2572 flags.go:64] FLAG: --enable-server="true" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843666 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843670 2572 flags.go:64] FLAG: --event-burst="100" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843673 2572 flags.go:64] FLAG: --event-qps="50" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843676 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843679 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843682 2572 flags.go:64] FLAG: --eviction-hard="" Nov 03 06:58:18.848813 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843686 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843689 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843692 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843695 2572 flags.go:64] FLAG: --eviction-soft="" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843698 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843701 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843704 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843707 2572 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843710 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843713 2572 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843716 2572 flags.go:64] FLAG: --feature-gates="" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843720 2572 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843723 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843726 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843729 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843732 2572 flags.go:64] FLAG: --healthz-port="10248" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843737 2572 flags.go:64] FLAG: --help="false" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843739 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843743 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843746 2572 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843749 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843752 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843755 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 06:58:18.849492 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843758 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843761 2572 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843764 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843767 2572 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843770 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843773 2572 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843776 2572 flags.go:64] FLAG: --kube-reserved="" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843779 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843782 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843785 2572 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843788 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843790 2572 flags.go:64] FLAG: --lock-file="" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843794 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843796 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843799 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843805 2572 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843807 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843811 2572 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843813 2572 flags.go:64] FLAG: --logging-format="text" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843816 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843819 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843822 2572 flags.go:64] FLAG: --manifest-url="" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843825 2572 flags.go:64] FLAG: --manifest-url-header="" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843829 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843832 2572 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 06:58:18.850090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843837 2572 flags.go:64] FLAG: --max-pods="110" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843840 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843843 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843846 2572 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843849 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843852 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843855 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843857 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843864 2572 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843868 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843872 2572 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843875 2572 flags.go:64] FLAG: --pod-cidr="" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843878 2572 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843881 2572 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843884 2572 flags.go:64] FLAG: --pods-per-core="0" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843887 2572 flags.go:64] FLAG: --port="10250" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843890 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843892 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0adb07709ae12134a" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843895 2572 flags.go:64] FLAG: --qos-reserved="" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843898 2572 flags.go:64] FLAG: --read-only-port="10255" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843901 2572 flags.go:64] FLAG: --register-node="true" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843904 2572 flags.go:64] FLAG: --register-with-taints="" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843907 2572 flags.go:64] FLAG: --registry-burst="10" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843910 2572 flags.go:64] FLAG: --registry-qps="5" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843913 2572 flags.go:64] FLAG: --reserved-cpus="" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843916 2572 flags.go:64] FLAG: --reserved-memory="" Nov 03 06:58:18.850661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843920 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843923 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843926 2572 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843928 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843931 2572 flags.go:64] FLAG: --runonce="false" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843934 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843937 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843940 2572 flags.go:64] FLAG: --seccomp-default="false" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843943 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843946 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843949 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843952 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843955 2572 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843958 2572 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843960 2572 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843965 2572 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843968 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843971 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843974 2572 flags.go:64] FLAG: --system-cgroups="" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843976 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843982 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843985 2572 flags.go:64] FLAG: --tls-cert-file="" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.843988 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844154 2572 flags.go:64] FLAG: --tls-min-version="" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844161 2572 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 06:58:18.851274 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844166 2572 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844172 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844177 2572 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844183 2572 flags.go:64] FLAG: --v="2" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844190 2572 flags.go:64] FLAG: --version="false" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844196 2572 flags.go:64] FLAG: --vmodule="" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844208 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.844224 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844642 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844649 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844652 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844656 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844662 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844666 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844669 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844672 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844675 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844678 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844680 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844684 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844687 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.851891 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844689 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844694 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844696 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844699 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844705 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844709 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844713 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844716 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844720 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844724 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844727 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844730 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844733 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844735 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844738 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844741 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844747 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844750 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844752 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.852392 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844756 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844759 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844761 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844764 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844767 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844770 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844773 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844776 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844778 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844784 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844786 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844789 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844792 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844794 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844798 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844801 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844803 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844806 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844809 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.852872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844812 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844814 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844817 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844823 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844826 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844829 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844831 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844834 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844837 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844839 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844842 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844845 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844847 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844850 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844853 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844856 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844861 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844864 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844868 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.853350 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844870 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844873 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844876 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844878 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844881 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844883 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844886 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844889 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844893 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844898 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844901 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844904 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844906 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844909 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844912 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844914 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844917 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844920 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844922 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844925 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.853796 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844928 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844930 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844935 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844938 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844941 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844943 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844946 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844948 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844952 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844954 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844957 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844985 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844989 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844993 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.844996 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845002 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.854369 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.845055 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845125 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845130 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845134 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845138 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845146 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845149 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845151 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845154 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845157 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845160 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845162 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845165 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845168 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845170 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845173 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845176 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845179 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845184 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.854755 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845187 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845189 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845192 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845194 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845197 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845200 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845203 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845206 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845210 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845213 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845216 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845222 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845225 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845227 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845230 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845232 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845235 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845237 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845241 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845244 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.855202 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845263 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845267 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845270 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845273 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845278 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845289 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845294 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845296 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845299 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845302 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845304 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845307 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845309 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845312 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845315 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845318 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845321 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845326 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845329 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845332 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.855715 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845336 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845339 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845341 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845344 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845346 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845349 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845352 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845354 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845357 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845360 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845366 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845371 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845373 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845376 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845378 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845382 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845387 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845390 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845394 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.856183 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845397 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845400 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845406 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845408 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845411 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845414 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845416 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845419 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845421 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845424 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845427 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845430 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845432 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845436 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845439 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845444 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845447 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845450 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845453 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.856631 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845455 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845458 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845460 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845463 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845466 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845468 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845471 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845474 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845479 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.845481 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.852888 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.852903 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.853576 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.853583 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853643 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853648 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853651 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853654 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853657 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.857085 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853660 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853662 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853665 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853667 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853670 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853672 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853675 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853678 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853680 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853683 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853686 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853688 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853691 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853693 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853696 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853699 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853701 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853705 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.857530 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853707 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853710 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853712 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853715 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853717 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853721 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853723 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853726 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853728 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853730 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853733 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853735 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853738 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853740 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853743 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853746 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853749 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853751 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853753 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853756 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.857949 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853758 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853761 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853763 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853766 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853768 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853771 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853773 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853776 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853778 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853781 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853784 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853786 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853788 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853791 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853794 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853797 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853800 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853803 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853805 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853808 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.858470 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853811 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853813 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853816 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853818 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853821 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853823 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853826 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853830 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853834 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853836 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853839 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853841 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853844 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853847 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853851 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853853 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853856 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853858 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853861 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.858921 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853863 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853866 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853868 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853871 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853874 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853876 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853879 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853882 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853884 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853887 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853890 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853892 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853895 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853897 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853900 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853902 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853904 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853907 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853910 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853913 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.859360 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853915 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853918 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853920 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.853923 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.853950 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854108 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854113 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854116 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854119 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854121 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854124 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854126 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854129 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854131 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854134 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854136 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854139 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.859809 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854142 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854145 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854148 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854152 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854156 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854160 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854163 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854165 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854168 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854170 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854173 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854175 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854178 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854181 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854184 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854186 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854189 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854191 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854194 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.860228 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854196 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854199 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854201 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854203 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854206 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854208 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854211 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854213 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854216 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854219 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854221 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854224 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854226 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854229 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854232 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854235 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854237 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854240 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854242 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.860650 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854245 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854247 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854250 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854252 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854255 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854257 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854259 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854262 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854264 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854266 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854269 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854271 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854274 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854276 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854279 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854281 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854284 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854286 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854288 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854291 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.861109 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854293 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854296 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854298 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854301 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854303 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854305 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854308 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854311 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854313 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854316 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854318 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854321 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854323 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854325 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854328 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854330 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854333 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854335 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854337 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.861609 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854340 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854342 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854345 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854347 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854349 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854352 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854355 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854357 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854360 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854363 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854367 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854370 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854372 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854375 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854378 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854381 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.862130 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854383 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.854407 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854439 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854442 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854446 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854448 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854455 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854458 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854461 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854463 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854466 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854468 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854471 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854473 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854475 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854478 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:18.862503 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854480 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854483 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854485 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854488 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854490 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854495 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854499 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854502 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854508 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854511 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854513 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854516 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854519 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854521 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854524 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854527 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854530 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854533 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854535 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:18.862872 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854538 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854540 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854543 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854545 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854548 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854551 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854553 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854556 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854558 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854561 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854563 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854566 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854568 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854571 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854573 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854576 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854578 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854580 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854583 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:18.863326 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854586 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854589 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854593 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854595 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854598 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854600 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854603 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854605 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854607 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854610 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854612 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854615 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854617 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854620 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854622 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854625 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854627 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854630 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854633 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854635 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:18.863772 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854638 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854640 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854643 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854645 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854648 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854650 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854653 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854655 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854657 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854660 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854662 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854665 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854667 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854670 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854674 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854676 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854679 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854682 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854684 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854687 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:18.864246 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854689 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854692 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854694 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854697 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854699 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854701 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854704 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854706 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854709 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854711 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854716 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854720 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854723 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:18.854725 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.854775 2572 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.857117 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.858167 2572 server.go:1014] "Starting client certificate rotation" Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.858265 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 06:58:18.864742 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.858312 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 06:58:18.865172 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.861048 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:58:18.865172 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.864111 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:58:18.874357 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.874337 2572 log.go:25] "Validated CRI v1 runtime API" Nov 03 06:58:18.878473 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.878458 2572 log.go:25] "Validated CRI v1 image API" Nov 03 06:58:18.879566 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.879529 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 06:58:18.882340 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.882312 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80006ed6-ff77-4f95-873f-129682c0524f:/dev/nvme0n1p3 a61abdef-5501-49b6-a028-ba81258537d9:/dev/nvme0n1p4] Nov 03 06:58:18.882425 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.882340 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 06:58:18.888468 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.888345 2572 manager.go:217] Machine: {Timestamp:2026-11-03 06:58:18.886944194 +0000 UTC m=+0.574835535 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101469 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27927827b97665a72aeeb56de8f254 SystemUUID:ec279278-27b9-7665-a72a-eeb56de8f254 BootID:e8f9580c-58ab-4870-b942-2b040345e9d7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:b1:50:73:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:b1:50:73:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:b1:27:d6:8e:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 06:58:18.888468 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.888461 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 06:58:18.888596 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.888534 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 06:58:18.891995 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.891972 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 06:58:18.892134 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.891997 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-184.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 06:58:18.892183 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.892137 2572 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 06:58:18.892183 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.892145 2572 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 06:58:18.892183 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.892182 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:58:18.892255 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.892201 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:58:18.893409 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.893398 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 06:58:18.898020 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.897995 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 06:58:18.898162 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.898152 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 06:58:18.898976 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.898967 2572 kubelet.go:493] "Attempting to sync node with API server" Nov 03 06:58:18.899006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.898980 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 06:58:18.899006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.898992 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 06:58:18.899006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.899000 2572 kubelet.go:405] "Adding apiserver pod source" Nov 03 06:58:18.899125 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.899024 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 06:58:18.899358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.899345 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:58:18.901648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.901631 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 06:58:18.908837 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.908825 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:58:18.909627 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909614 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 06:58:18.909661 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909646 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 06:58:18.909791 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909782 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 06:58:18.909819 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909797 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 06:58:18.909819 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909804 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 06:58:18.909819 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909809 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 06:58:18.909896 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909822 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 06:58:18.909896 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.909829 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 06:58:18.910480 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.910471 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 06:58:18.910510 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.910480 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 06:58:18.910510 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.910488 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 06:58:18.910510 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.910494 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 06:58:18.910510 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.910508 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 06:58:18.910631 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.910516 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 06:58:18.912030 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.912020 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 06:58:18.912062 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.912032 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 06:58:18.915828 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.915817 2572 server.go:1263] "Started kubelet" Nov 03 06:58:18.915926 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.915872 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 06:58:18.916404 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.916354 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 06:58:18.916452 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.916428 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 06:58:18.916683 ip-10-0-136-184 systemd[1]: Started Kubernetes Kubelet. Nov 03 06:58:18.917940 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.917906 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 06:58:18.918471 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.918457 2572 server.go:317] "Adding debug handlers to kubelet server" Nov 03 06:58:18.922677 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:18.922661 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 06:58:18.925118 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.925099 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 06:58:18.925718 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.925704 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 06:58:18.925792 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:18.925773 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:18.926556 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926473 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 06:58:18.926556 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926489 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 06:58:18.926556 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926478 2572 factory.go:55] Registering systemd factory Nov 03 06:58:18.926705 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926572 2572 factory.go:223] Registration of the systemd container factory successfully Nov 03 06:58:18.926705 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926683 2572 reconstruct.go:97] "Volume reconstruction finished" Nov 03 06:58:18.926705 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926691 2572 reconciler.go:29] "Reconciler: start to sync state" Nov 03 06:58:18.926878 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926862 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 06:58:18.926991 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926977 2572 factory.go:153] Registering CRI-O factory Nov 03 06:58:18.926991 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.926991 2572 factory.go:223] Registration of the crio container factory successfully Nov 03 06:58:18.927097 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.927066 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 06:58:18.927097 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.927087 2572 factory.go:103] Registering Raw factory Nov 03 06:58:18.927097 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.927097 2572 manager.go:1199] Started watching for new ooms in manager Nov 03 06:58:18.928305 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.928291 2572 manager.go:319] Starting recovery of all containers Nov 03 06:58:18.939109 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.939093 2572 manager.go:324] Recovery completed Nov 03 06:58:18.942904 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.942890 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:18.944817 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.944792 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:18.944873 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.944831 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:18.944873 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.944841 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:18.945340 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.945326 2572 cpu_manager.go:226] "Starting" policy="none" Nov 03 06:58:18.945340 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.945337 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 06:58:18.945340 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.945335 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:18.945497 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.945352 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 06:58:18.946183 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.946168 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:18.946236 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.946194 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:18.946236 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.946204 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:18.947449 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.947438 2572 policy_none.go:50] "Start" Nov 03 06:58:18.947493 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.947453 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 06:58:18.947493 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.947463 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 06:58:18.949427 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.949417 2572 policy_none.go:44] "Start" Nov 03 06:58:18.981916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.981796 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 06:58:18.985928 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.985917 2572 manager.go:341] "Starting Device Plugin manager" Nov 03 06:58:18.985985 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:18.985960 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 06:58:18.985985 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.985971 2572 server.go:87] "Starting device plugin registration server" Nov 03 06:58:18.986190 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.986180 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 06:58:18.986222 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.986192 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 06:58:18.986294 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.986278 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 06:58:18.986384 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.986363 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 06:58:18.986384 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:18.986379 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 06:58:18.986902 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:18.986873 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 06:58:18.986980 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:18.986935 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:19.061410 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.061384 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-184.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 06:58:19.063275 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.061418 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-184.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 06:58:19.063275 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.061361 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-184.us-west-1.compute.internal.18e476bb6a5dbe8a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-184.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-184.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:18.915790474 +0000 UTC m=+0.603681812,LastTimestamp:2026-11-03 06:58:18.915790474 +0000 UTC m=+0.603681812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-184.us-west-1.compute.internal,}" Nov 03 06:58:19.072867 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.072843 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 06:58:19.074151 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.074135 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 06:58:19.074202 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.074160 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 06:58:19.074202 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.074179 2572 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 06:58:19.074280 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.074215 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 06:58:19.087208 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.087166 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:19.088291 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.088276 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:19.088337 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.088309 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:19.088337 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.088325 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:19.088395 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.088348 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.117628 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.117604 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-184.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 06:58:19.118307 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.117621 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-184.us-west-1.compute.internal.18e476bb6c18a9c3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-184.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-184.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-184.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:18.944817603 +0000 UTC m=+0.632708944,LastTimestamp:2026-11-03 06:58:18.944817603 +0000 UTC m=+0.632708944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-184.us-west-1.compute.internal,}" Nov 03 06:58:19.151031 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.151001 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 06:58:19.151106 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.151067 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mdvhj" Nov 03 06:58:19.174929 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.174906 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal"] Nov 03 06:58:19.175001 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.174958 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:19.175740 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.175726 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:19.175790 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.175753 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:19.175790 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.175764 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:19.177871 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.177858 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:19.178023 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.177995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.178059 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.178047 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:19.178548 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.178531 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:19.178548 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.178541 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:19.178620 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.178564 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:19.178620 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.178566 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:19.178620 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.178588 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:19.178620 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.178577 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:19.180648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.180635 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.180688 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.180661 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:19.181317 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.181301 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:19.181390 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.181330 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:19.181390 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.181344 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:19.207435 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.207421 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.210102 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.210087 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.229036 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.229003 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.229107 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.229046 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.229107 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.229085 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f565955bc42ade3fa3b9ef6d6540a53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"1f565955bc42ade3fa3b9ef6d6540a53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.329934 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.329907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.329934 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.329933 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.330114 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.329951 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f565955bc42ade3fa3b9ef6d6540a53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"1f565955bc42ade3fa3b9ef6d6540a53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.330114 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.329993 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f565955bc42ade3fa3b9ef6d6540a53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"1f565955bc42ade3fa3b9ef6d6540a53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.330114 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.330032 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.330114 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.330052 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.334403 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.334387 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:19.334403 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.334393 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:19.335758 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.335741 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:19.336079 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.336049 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.336079 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.336083 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-184.us-west-1.compute.internal\": node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:19.450605 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.450539 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:19.509930 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.509909 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.512844 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.512828 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:19.550972 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.550940 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:19.651084 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.651053 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:19.751184 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.751125 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:19.851401 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.851375 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:19.857544 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.857519 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 06:58:19.857708 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.857681 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 06:58:19.857747 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.857697 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 06:58:19.857747 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:19.857681 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 06:58:19.857747 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.857654 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa5d097cf21004010b37820ad9a97310-f5a8f4c20a3cd5e4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-184.us-west-1.compute.internal.18e476bb6c18f08e\": read tcp 10.0.136.184:34740->52.14.173.122:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-184.us-west-1.compute.internal.18e476bb6c18f08e default 4759 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-184.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-184.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-184.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:18 +0000 UTC,LastTimestamp:2026-11-03 06:58:19.088316952 +0000 UTC m=+0.776208290,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-184.us-west-1.compute.internal,}" Nov 03 06:58:19.952363 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:19.952339 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.053048 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:20.052979 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.153225 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:20.153199 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.155360 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.155336 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 06:53:19 +0000 UTC" deadline="2028-06-13 10:22:48.831672587 +0000 UTC" Nov 03 06:58:20.155414 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.155361 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14115h24m28.676315037s" Nov 03 06:58:20.162493 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.162479 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:20.163385 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.163370 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:20.163464 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.163404 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:20.163464 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.163421 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:20.253465 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:20.253443 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.353664 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:20.353604 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.453976 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:20.453951 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.554511 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:20.554481 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.583524 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:20.583493 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15c975752830b5906436794da287b80.slice/crio-cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32 WatchSource:0}: Error finding container cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32: Status 404 returned error can't find the container with id cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32 Nov 03 06:58:20.584039 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:20.584004 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f565955bc42ade3fa3b9ef6d6540a53.slice/crio-6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce WatchSource:0}: Error finding container 6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce: Status 404 returned error can't find the container with id 6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce Nov 03 06:58:20.589163 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.589151 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:58:20.654621 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:20.654570 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:58:20.684042 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.684024 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:20.726144 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.726112 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:20.731259 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.731239 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:20.803098 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.803074 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:58:20.803751 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.803737 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:58:20.869285 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.869260 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:58:20.900626 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.900589 2572 apiserver.go:52] "Watching apiserver" Nov 03 06:58:20.927042 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:20.926969 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 06:58:21.071471 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.071434 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 06:58:21.071745 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.071727 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zq5qq","openshift-multus/multus-additional-cni-plugins-x5cxd","openshift-multus/network-metrics-daemon-g8m65","kube-system/global-pull-secret-syncer-dtzxb","openshift-cluster-node-tuning-operator/tuned-zhjnj","openshift-network-diagnostics/network-check-target-dfpc7","kube-system/konnectivity-agent-4gwnd","openshift-image-registry/node-ca-9vk5q","openshift-multus/multus-v9wll","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-vn65v","openshift-ovn-kubernetes/ovnkube-node-pb8jn","kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb"] Nov 03 06:58:21.075765 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.075747 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.078060 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.078039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.080382 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.080360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:21.080481 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.080444 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:21.083577 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.083044 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.085742 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.085703 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.088527 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.088502 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:21.088621 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.088575 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:21.090929 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.090913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.093311 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.093295 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.096175 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.096160 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.096261 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.096240 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:21.098602 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.098580 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.100980 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.100960 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.103279 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.103234 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"c15c975752830b5906436794da287b80","Type":"ContainerStarted","Data":"cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32"} Nov 03 06:58:21.103365 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.103290 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"1f565955bc42ade3fa3b9ef6d6540a53","Type":"ContainerStarted","Data":"6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce"} Nov 03 06:58:21.103365 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.103316 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.127701 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.127684 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 06:58:21.137212 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.137180 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 06:58:21.137420 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.137399 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 06:58:21.137482 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.137436 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 06:58:21.137482 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.137407 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 06:58:21.137482 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.137406 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 06:58:21.137621 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.137512 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 06:58:21.137700 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.137686 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5znfn\"" Nov 03 06:58:21.138228 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.138210 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44mbd\"" Nov 03 06:58:21.139332 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139316 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-log-socket\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.139374 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-host\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.139374 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139365 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-kubelet\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.139459 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-node-log\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.139459 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139398 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6eb01e1-380a-468c-8097-d34540808454-ovn-node-metrics-cert\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.139557 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139466 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/e6eb01e1-380a-468c-8097-d34540808454-kube-api-access-tc9m9\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.139557 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7781680-4a88-4a3f-8fc1-61c702b89f63-agent-certs\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.139633 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-multus\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.139633 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139606 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-etc-kubernetes\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.139736 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysconfig\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.139736 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139687 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffvz\" (UniqueName: \"kubernetes.io/projected/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-kube-api-access-sffvz\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.139736 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139713 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-multus-daemon-config\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.139846 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.139846 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-socket-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.139913 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139846 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-modprobe-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.139913 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.139913 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-dbus\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.139999 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-kubelet\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.139999 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.139999 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.139973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-run\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.140171 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140001 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:21.140171 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-host-slash\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.140171 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-systemd-units\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140171 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140067 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-hostroot\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140171 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-k8s-cni-cncf-io\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140171 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-conf-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140174 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-slash\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140227 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-netns\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140255 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-netd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140286 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140318 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjv9\" (UniqueName: \"kubernetes.io/projected/0dbc159e-0969-417d-91fb-819d526579a3-kube-api-access-qtjv9\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140336 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqdw\" (UniqueName: \"kubernetes.io/projected/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kube-api-access-ckqdw\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.140416 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-hosts-file\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glrf\" (UniqueName: \"kubernetes.io/projected/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-kube-api-access-2glrf\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-cni-binary-copy\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-netns\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-system-cni-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-device-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140520 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfgw\" (UniqueName: \"kubernetes.io/projected/580a0715-a63d-43ef-983c-6d459af6ba42-kube-api-access-wxfgw\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140533 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-systemd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-bin\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-script-lib\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-cnibin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-bin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-conf\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140650 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-lib-modules\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140664 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7781680-4a88-4a3f-8fc1-61c702b89f63-konnectivity-ca\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-socket-dir-parent\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.140796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140759 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-multus-certs\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-registration-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140805 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-systemd\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-etc-tuned\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140834 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-system-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140920 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-kubernetes\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.140986 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2xf\" (UniqueName: \"kubernetes.io/projected/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-kube-api-access-cj2xf\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141003 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-ovn\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-host\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-kubelet-config\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141092 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-sys\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.141447 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-cnibin\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-tmp\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-var-lib-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-config\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141891 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-env-overrides\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-sys-fs\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.141994 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-var-lib-kubelet\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-serviceca\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-tmp-dir\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142148 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh76m\" (UniqueName: \"kubernetes.io/projected/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-kube-api-access-bh76m\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.142241 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4f7r\" (UniqueName: \"kubernetes.io/projected/95020920-a029-4c17-8ce1-e66a4982a00d-kube-api-access-p4f7r\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.142708 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-iptables-alerter-script\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.142708 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142286 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-etc-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.142708 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.142306 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9lht\"" Nov 03 06:58:21.145056 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.145040 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:21.145140 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.145087 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:21.145351 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.145330 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dmgt\"" Nov 03 06:58:21.150030 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.149840 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxfbc\"" Nov 03 06:58:21.186028 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.185959 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-htlhr\"" Nov 03 06:58:21.186028 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.185960 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8c889\"" Nov 03 06:58:21.186742 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.186718 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:21.186980 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.186964 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 06:58:21.187064 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.187043 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 06:58:21.187064 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.187055 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 06:58:21.187167 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.187059 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 06:58:21.190636 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.190622 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 06:58:21.190696 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.190661 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 06:58:21.190696 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.190661 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-br79x\"" Nov 03 06:58:21.192363 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.192350 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 06:58:21.192409 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.192351 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 06:58:21.192486 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.192473 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:21.192558 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.192537 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 06:58:21.192602 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.192587 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 06:58:21.192681 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.192665 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 06:58:21.197584 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.197570 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 06:58:21.204711 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.204695 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 06:58:21.204863 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.204849 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 06:58:21.204921 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.204852 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lr4sn\"" Nov 03 06:58:21.204921 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.204866 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 06:58:21.204921 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.204899 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r48n5" Nov 03 06:58:21.243407 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-modprobe-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.243465 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243411 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.243465 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-dbus\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.243465 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-kubelet\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.243610 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.243610 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243549 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-kubelet\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.243610 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-run\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.243610 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243576 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.243610 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243584 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-dbus\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.243610 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243576 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-modprobe-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243633 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-run\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243617 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-host-slash\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-systemd-units\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243669 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-host-slash\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-hostroot\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-k8s-cni-cncf-io\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.243709 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-conf-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243738 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243743 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-k8s-cni-cncf-io\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243759 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-slash\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243739 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-hostroot\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243797 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-slash\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.243808 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:21.74376108 +0000 UTC m=+3.431652425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:21.243916 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243708 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-systemd-units\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243782 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-conf-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243852 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-netns\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243880 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-netd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243904 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-netns\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjv9\" (UniqueName: \"kubernetes.io/projected/0dbc159e-0969-417d-91fb-819d526579a3-kube-api-access-qtjv9\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243945 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-netd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqdw\" (UniqueName: \"kubernetes.io/projected/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kube-api-access-ckqdw\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.243989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.244096 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-hosts-file\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.244142 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret podName:ca92ce4d-7cf4-4da5-8609-920db260c2d3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:21.744126582 +0000 UTC m=+3.432017913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret") pod "global-pull-secret-syncer-dtzxb" (UID: "ca92ce4d-7cf4-4da5-8609-920db260c2d3") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glrf\" (UniqueName: \"kubernetes.io/projected/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-kube-api-access-2glrf\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244184 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-hosts-file\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-cni-binary-copy\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.244799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244225 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-netns\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244280 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-netns\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-system-cni-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-device-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244349 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfgw\" (UniqueName: \"kubernetes.io/projected/580a0715-a63d-43ef-983c-6d459af6ba42-kube-api-access-wxfgw\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244385 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-system-cni-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-systemd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244430 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-device-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244438 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-systemd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244439 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-bin\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-script-lib\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244474 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-bin\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-cnibin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244512 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-bin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-conf\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-cnibin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.245629 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-lib-modules\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244588 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7781680-4a88-4a3f-8fc1-61c702b89f63-konnectivity-ca\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244599 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-bin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-socket-dir-parent\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-multus-certs\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-registration-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-systemd\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-etc-tuned\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244718 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-multus-certs\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244720 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-socket-dir-parent\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244735 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-lib-modules\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244678 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-conf\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244728 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-cni-binary-copy\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244780 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-registration-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244808 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-systemd\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244924 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.246598 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244919 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-system-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.244991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-system-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245074 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-kubernetes\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245102 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-script-lib\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2xf\" (UniqueName: \"kubernetes.io/projected/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-kube-api-access-cj2xf\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245164 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245175 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-ovn\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245184 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-kubernetes\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-host\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245205 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245215 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245228 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-kubelet-config\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245249 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-ovn\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-sys\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245261 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-host\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.247443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245258 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7781680-4a88-4a3f-8fc1-61c702b89f63-konnectivity-ca\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245315 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-sys\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245322 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-kubelet-config\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245351 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-cnibin\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245391 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-tmp\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-var-lib-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245440 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-cnibin\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-config\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-env-overrides\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-sys-fs\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245512 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-var-lib-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245527 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-var-lib-kubelet\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-serviceca\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245561 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-sys-fs\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-tmp-dir\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.248358 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245617 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-var-lib-kubelet\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh76m\" (UniqueName: \"kubernetes.io/projected/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-kube-api-access-bh76m\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245663 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f7r\" (UniqueName: \"kubernetes.io/projected/95020920-a029-4c17-8ce1-e66a4982a00d-kube-api-access-p4f7r\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-iptables-alerter-script\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-etc-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-log-socket\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245834 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-host\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-kubelet\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-node-log\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6eb01e1-380a-468c-8097-d34540808454-ovn-node-metrics-cert\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245956 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.245996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/e6eb01e1-380a-468c-8097-d34540808454-kube-api-access-tc9m9\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7781680-4a88-4a3f-8fc1-61c702b89f63-agent-certs\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246037 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-log-socket\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249013 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-multus\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246085 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-node-log\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246098 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-tmp-dir\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-etc-kubernetes\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246126 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-etc-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysconfig\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246177 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-host\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246183 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-kubelet\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffvz\" (UniqueName: \"kubernetes.io/projected/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-kube-api-access-sffvz\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-multus-daemon-config\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246335 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-socket-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246548 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-socket-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246570 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-iptables-alerter-script\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246631 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-multus\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-etc-kubernetes\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.246724 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysconfig\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.247237 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.249504 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.247319 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-multus-daemon-config\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.250006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.247448 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-serviceca\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.250006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.247523 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-config\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.250006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.247660 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-env-overrides\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.250006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.248866 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-etc-tuned\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.250006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.248906 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-tmp\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.250006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.249166 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6eb01e1-380a-468c-8097-d34540808454-ovn-node-metrics-cert\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.250006 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.249262 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7781680-4a88-4a3f-8fc1-61c702b89f63-agent-certs\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.317801 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.317768 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:21.317801 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.317798 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:21.318189 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.317813 2572 projected.go:196] Error preparing data for projected volume kube-api-access-gc428 for pod openshift-network-diagnostics/network-check-target-dfpc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:21.318189 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.317869 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428 podName:e47cc873-c3fe-421e-8882-9ce4b4ff3737 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:21.81785041 +0000 UTC m=+3.505741768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gc428" (UniqueName: "kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428") pod "network-check-target-dfpc7" (UID: "e47cc873-c3fe-421e-8882-9ce4b4ff3737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:21.319614 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.319587 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2xf\" (UniqueName: \"kubernetes.io/projected/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-kube-api-access-cj2xf\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:21.319783 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.319749 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqdw\" (UniqueName: \"kubernetes.io/projected/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kube-api-access-ckqdw\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.319896 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.319756 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glrf\" (UniqueName: \"kubernetes.io/projected/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-kube-api-access-2glrf\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.320208 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.320187 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/e6eb01e1-380a-468c-8097-d34540808454-kube-api-access-tc9m9\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.320281 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.320186 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh76m\" (UniqueName: \"kubernetes.io/projected/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-kube-api-access-bh76m\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.320626 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.320609 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4f7r\" (UniqueName: \"kubernetes.io/projected/95020920-a029-4c17-8ce1-e66a4982a00d-kube-api-access-p4f7r\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.321293 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.321272 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjv9\" (UniqueName: \"kubernetes.io/projected/0dbc159e-0969-417d-91fb-819d526579a3-kube-api-access-qtjv9\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.321895 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.321873 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffvz\" (UniqueName: \"kubernetes.io/projected/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-kube-api-access-sffvz\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.322823 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.322802 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfgw\" (UniqueName: \"kubernetes.io/projected/580a0715-a63d-43ef-983c-6d459af6ba42-kube-api-access-wxfgw\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.350326 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.350302 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:21.390353 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.390327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v9wll" Nov 03 06:58:21.400649 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.400627 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580a0715_a63d_43ef_983c_6d459af6ba42.slice/crio-3d06ad07fb20cba3df1f1a5d88db3399ffa319a36a76708a1a30268825e90267 WatchSource:0}: Error finding container 3d06ad07fb20cba3df1f1a5d88db3399ffa319a36a76708a1a30268825e90267: Status 404 returned error can't find the container with id 3d06ad07fb20cba3df1f1a5d88db3399ffa319a36a76708a1a30268825e90267 Nov 03 06:58:21.407409 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.407391 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:58:21.414371 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.414353 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:58:21.416211 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.416189 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95020920_a029_4c17_8ce1_e66a4982a00d.slice/crio-1ba57ec5310c1c34639a5da44717d3431f9254b5069d2c3b9527ee3144d26458 WatchSource:0}: Error finding container 1ba57ec5310c1c34639a5da44717d3431f9254b5069d2c3b9527ee3144d26458: Status 404 returned error can't find the container with id 1ba57ec5310c1c34639a5da44717d3431f9254b5069d2c3b9527ee3144d26458 Nov 03 06:58:21.417311 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.417298 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:58:21.419702 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.419661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:21.422172 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.422149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:58:21.423889 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.423868 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbc159e_0969_417d_91fb_819d526579a3.slice/crio-d94e5adca559e4458bbef324a1d9b3f18209be7f3c4142fc14adb36715755231 WatchSource:0}: Error finding container d94e5adca559e4458bbef324a1d9b3f18209be7f3c4142fc14adb36715755231: Status 404 returned error can't find the container with id d94e5adca559e4458bbef324a1d9b3f18209be7f3c4142fc14adb36715755231 Nov 03 06:58:21.425254 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.425180 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:58:21.427647 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.427622 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e402a1_d12e_4559_aae4_38ca72e7f1cf.slice/crio-78c3063a3ea8935541cf2ba5c4bbb74c0e17ff83679aae44b1a2c200e577721d WatchSource:0}: Error finding container 78c3063a3ea8935541cf2ba5c4bbb74c0e17ff83679aae44b1a2c200e577721d: Status 404 returned error can't find the container with id 78c3063a3ea8935541cf2ba5c4bbb74c0e17ff83679aae44b1a2c200e577721d Nov 03 06:58:21.433705 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.433680 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7781680_4a88_4a3f_8fc1_61c702b89f63.slice/crio-30fd3627b9c4432586acd0a7ebc07375d416be91d21f2fab256d2a4ad5c62f91 WatchSource:0}: Error finding container 30fd3627b9c4432586acd0a7ebc07375d416be91d21f2fab256d2a4ad5c62f91: Status 404 returned error can't find the container with id 30fd3627b9c4432586acd0a7ebc07375d416be91d21f2fab256d2a4ad5c62f91 Nov 03 06:58:21.434381 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.434356 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc2a19a_80f8_4318_a6fd_b246b3022d0e.slice/crio-63abe62800a3852c2c711306dc0100fb3397a7ba6cf9d0280f3bfd0943d1c591 WatchSource:0}: Error finding container 63abe62800a3852c2c711306dc0100fb3397a7ba6cf9d0280f3bfd0943d1c591: Status 404 returned error can't find the container with id 63abe62800a3852c2c711306dc0100fb3397a7ba6cf9d0280f3bfd0943d1c591 Nov 03 06:58:21.437334 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.437308 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b39d84_7483_46fb_ab4b_d8de4e3a1212.slice/crio-5bc1bf10904b637729a1dec0e044c79a0e3050e11d3d3a258df22ef86e5cac59 WatchSource:0}: Error finding container 5bc1bf10904b637729a1dec0e044c79a0e3050e11d3d3a258df22ef86e5cac59: Status 404 returned error can't find the container with id 5bc1bf10904b637729a1dec0e044c79a0e3050e11d3d3a258df22ef86e5cac59 Nov 03 06:58:21.440587 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.440569 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:58:21.448838 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.448818 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:21.458902 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.458881 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:21.459535 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:21.459513 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6eb01e1_380a_468c_8097_d34540808454.slice/crio-724b5a7fe3285e73476c8ae0592a9cec7b229b3af70609c0e384cd3bc420c87c WatchSource:0}: Error finding container 724b5a7fe3285e73476c8ae0592a9cec7b229b3af70609c0e384cd3bc420c87c: Status 404 returned error can't find the container with id 724b5a7fe3285e73476c8ae0592a9cec7b229b3af70609c0e384cd3bc420c87c Nov 03 06:58:21.749757 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.749656 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:21.749757 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.749722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:21.749985 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.749836 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:21.749985 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.749864 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:21.749985 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.749909 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:22.749887243 +0000 UTC m=+4.437778586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:21.749985 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.749932 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret podName:ca92ce4d-7cf4-4da5-8609-920db260c2d3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:22.749920476 +0000 UTC m=+4.437811808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret") pod "global-pull-secret-syncer-dtzxb" (UID: "ca92ce4d-7cf4-4da5-8609-920db260c2d3") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:21.850814 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:21.850775 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:21.851068 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.851037 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:21.851161 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.851086 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:21.851161 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.851101 2572 projected.go:196] Error preparing data for projected volume kube-api-access-gc428 for pod openshift-network-diagnostics/network-check-target-dfpc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:21.851161 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:21.851158 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428 podName:e47cc873-c3fe-421e-8882-9ce4b4ff3737 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:22.851140603 +0000 UTC m=+4.539031946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc428" (UniqueName: "kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428") pod "network-check-target-dfpc7" (UID: "e47cc873-c3fe-421e-8882-9ce4b4ff3737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:22.084459 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.084376 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" event={"ID":"10e402a1-d12e-4559-aae4-38ca72e7f1cf","Type":"ContainerStarted","Data":"78c3063a3ea8935541cf2ba5c4bbb74c0e17ff83679aae44b1a2c200e577721d"} Nov 03 06:58:22.085945 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.085914 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerStarted","Data":"d94e5adca559e4458bbef324a1d9b3f18209be7f3c4142fc14adb36715755231"} Nov 03 06:58:22.087355 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.087327 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9wll" event={"ID":"580a0715-a63d-43ef-983c-6d459af6ba42","Type":"ContainerStarted","Data":"3d06ad07fb20cba3df1f1a5d88db3399ffa319a36a76708a1a30268825e90267"} Nov 03 06:58:22.088740 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.088713 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"724b5a7fe3285e73476c8ae0592a9cec7b229b3af70609c0e384cd3bc420c87c"} Nov 03 06:58:22.090160 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.090130 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zq5qq" event={"ID":"eab2a8b5-c6f2-45df-9fcc-161d6ad92498","Type":"ContainerStarted","Data":"9193dc5965f783ebccd05cf612d68f63fc4044a31db970b5a26e18cdf78cfb7c"} Nov 03 06:58:22.091505 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.091480 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vn65v" event={"ID":"c6b39d84-7483-46fb-ab4b-d8de4e3a1212","Type":"ContainerStarted","Data":"5bc1bf10904b637729a1dec0e044c79a0e3050e11d3d3a258df22ef86e5cac59"} Nov 03 06:58:22.092744 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.092710 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9vk5q" event={"ID":"2fc2a19a-80f8-4318-a6fd-b246b3022d0e","Type":"ContainerStarted","Data":"63abe62800a3852c2c711306dc0100fb3397a7ba6cf9d0280f3bfd0943d1c591"} Nov 03 06:58:22.094124 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.094091 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4gwnd" event={"ID":"f7781680-4a88-4a3f-8fc1-61c702b89f63","Type":"ContainerStarted","Data":"30fd3627b9c4432586acd0a7ebc07375d416be91d21f2fab256d2a4ad5c62f91"} Nov 03 06:58:22.095349 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.095299 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" event={"ID":"95020920-a029-4c17-8ce1-e66a4982a00d","Type":"ContainerStarted","Data":"1ba57ec5310c1c34639a5da44717d3431f9254b5069d2c3b9527ee3144d26458"} Nov 03 06:58:22.208308 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.208243 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:21 +0000 UTC" deadline="2028-06-02 05:06:43.954640869 +0000 UTC" Nov 03 06:58:22.208308 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.208304 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13846h8m21.7463406s" Nov 03 06:58:22.758543 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.758131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:22.758543 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.758187 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:22.758543 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.758318 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:22.758543 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.758392 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret podName:ca92ce4d-7cf4-4da5-8609-920db260c2d3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:24.758373167 +0000 UTC m=+6.446264507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret") pod "global-pull-secret-syncer-dtzxb" (UID: "ca92ce4d-7cf4-4da5-8609-920db260c2d3") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:22.758543 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.758476 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:22.758543 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.758513 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:24.758502447 +0000 UTC m=+6.446393780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:22.860713 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:22.860429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:22.860713 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.860584 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:22.860713 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.860603 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:22.860713 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.860613 2572 projected.go:196] Error preparing data for projected volume kube-api-access-gc428 for pod openshift-network-diagnostics/network-check-target-dfpc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:22.860713 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:22.860663 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428 podName:e47cc873-c3fe-421e-8882-9ce4b4ff3737 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:24.860649532 +0000 UTC m=+6.548540859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc428" (UniqueName: "kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428") pod "network-check-target-dfpc7" (UID: "e47cc873-c3fe-421e-8882-9ce4b4ff3737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:23.077091 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:23.077023 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:23.077253 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:23.077190 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:23.077327 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:23.077269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:23.077384 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:23.077344 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:23.077437 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:23.077414 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:23.077496 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:23.077482 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:23.209328 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:23.209233 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:21 +0000 UTC" deadline="2028-06-07 11:58:39.529689773 +0000 UTC" Nov 03 06:58:23.209328 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:23.209277 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13973h0m16.320417278s" Nov 03 06:58:24.111579 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:24.111262 2572 generic.go:356] "Generic (PLEG): container finished" podID="c15c975752830b5906436794da287b80" containerID="ded14717958d4b1b1b4687ffd6b41dffaa142318c3bf4c973350d91aae8be084" exitCode=0 Nov 03 06:58:24.111579 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:24.111568 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"c15c975752830b5906436794da287b80","Type":"ContainerDied","Data":"ded14717958d4b1b1b4687ffd6b41dffaa142318c3bf4c973350d91aae8be084"} Nov 03 06:58:24.116194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:24.116163 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"1f565955bc42ade3fa3b9ef6d6540a53","Type":"ContainerStarted","Data":"ed1d537994cc17d0f766a467fc4b83670e786c952917a776991f34c5baddc1ab"} Nov 03 06:58:24.804453 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:24.803954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:24.804453 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:24.804005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:24.804453 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.804193 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:24.804453 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.804245 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret podName:ca92ce4d-7cf4-4da5-8609-920db260c2d3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:28.804228096 +0000 UTC m=+10.492119425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret") pod "global-pull-secret-syncer-dtzxb" (UID: "ca92ce4d-7cf4-4da5-8609-920db260c2d3") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:24.804453 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.804308 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:24.804453 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.804331 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:28.804323529 +0000 UTC m=+10.492214857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:24.905552 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:24.905177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:24.905552 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.905406 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:24.905552 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.905429 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:24.905552 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.905444 2572 projected.go:196] Error preparing data for projected volume kube-api-access-gc428 for pod openshift-network-diagnostics/network-check-target-dfpc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:24.905552 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:24.905511 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428 podName:e47cc873-c3fe-421e-8882-9ce4b4ff3737 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:28.905492759 +0000 UTC m=+10.593384091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc428" (UniqueName: "kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428") pod "network-check-target-dfpc7" (UID: "e47cc873-c3fe-421e-8882-9ce4b4ff3737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:25.078286 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:25.078210 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:25.078431 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:25.078355 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:25.078779 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:25.078756 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:25.078900 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:25.078881 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:25.078964 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:25.078939 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:25.079018 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:25.079006 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:25.142774 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:25.142738 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"c15c975752830b5906436794da287b80","Type":"ContainerStarted","Data":"11f92ef661280d5617390c852bb68938d3c1a260ea742d21ae3d4ea805882aee"} Nov 03 06:58:25.290573 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:25.290510 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" podStartSLOduration=4.290488479 podStartE2EDuration="4.290488479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:24.378406308 +0000 UTC m=+6.066297658" watchObservedRunningTime="2026-11-03 06:58:25.290488479 +0000 UTC m=+6.978379824" Nov 03 06:58:25.291393 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:25.291357 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" podStartSLOduration=4.291348744 podStartE2EDuration="4.291348744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:25.290376922 +0000 UTC m=+6.978268270" watchObservedRunningTime="2026-11-03 06:58:25.291348744 +0000 UTC m=+6.979240096" Nov 03 06:58:26.153176 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:26.151418 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="4d030c626ab7f8d5beae44262ef0c601ca2f37f2462f08cfc45f8c31f2d5892b" exitCode=0 Nov 03 06:58:26.153176 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:26.152839 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerDied","Data":"4d030c626ab7f8d5beae44262ef0c601ca2f37f2462f08cfc45f8c31f2d5892b"} Nov 03 06:58:27.080175 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:27.079328 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:27.080175 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:27.079457 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:27.080175 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:27.079853 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:27.080175 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:27.079949 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:27.080175 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:27.080016 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:27.080175 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:27.080129 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:27.157061 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:27.156866 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4gwnd" event={"ID":"f7781680-4a88-4a3f-8fc1-61c702b89f63","Type":"ContainerStarted","Data":"0e961b3dfc4138f7d4e1c986a4206c1a99e5a54fff92e71684719d1d3f156e44"} Nov 03 06:58:27.288140 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:27.287322 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4gwnd" podStartSLOduration=3.464061782 podStartE2EDuration="8.287300698s" totalImagesPullingTime="4.823238916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:27.287253921 +0000 UTC m=+8.975145272" watchObservedRunningTime="2026-11-03 06:58:27.287300698 +0000 UTC m=+8.975192050" Nov 03 06:58:28.162208 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:28.161940 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9vk5q" event={"ID":"2fc2a19a-80f8-4318-a6fd-b246b3022d0e","Type":"ContainerStarted","Data":"ffc343265086cf85694eee6c5b2142b38b9d0cb018d9918d387ac3b9dfa54c57"} Nov 03 06:58:28.167546 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:28.167515 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" event={"ID":"10e402a1-d12e-4559-aae4-38ca72e7f1cf","Type":"ContainerStarted","Data":"735c57577e3b53a0e333f8593ac697eff1aceaf9b1cdf8eb6a68e3592dc7a916"} Nov 03 06:58:28.280906 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:28.280846 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9vk5q" podStartSLOduration=3.358855155 podStartE2EDuration="9.280828911s" totalImagesPullingTime="5.921973756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:28.280777457 +0000 UTC m=+9.968668804" watchObservedRunningTime="2026-11-03 06:58:28.280828911 +0000 UTC m=+9.968720259" Nov 03 06:58:28.840957 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:28.840926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:28.841137 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:28.840970 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:28.841137 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.841119 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:28.841277 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.841171 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret podName:ca92ce4d-7cf4-4da5-8609-920db260c2d3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:36.841157754 +0000 UTC m=+18.529049087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret") pod "global-pull-secret-syncer-dtzxb" (UID: "ca92ce4d-7cf4-4da5-8609-920db260c2d3") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:28.841535 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.841518 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:28.841631 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.841564 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:36.841549448 +0000 UTC m=+18.529440775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:28.941851 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:28.941552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:28.941851 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.941662 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:28.941851 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.941677 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:28.941851 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.941685 2572 projected.go:196] Error preparing data for projected volume kube-api-access-gc428 for pod openshift-network-diagnostics/network-check-target-dfpc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:28.941851 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:28.941733 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428 podName:e47cc873-c3fe-421e-8882-9ce4b4ff3737 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:36.941716515 +0000 UTC m=+18.629607842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc428" (UniqueName: "kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428") pod "network-check-target-dfpc7" (UID: "e47cc873-c3fe-421e-8882-9ce4b4ff3737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:29.079705 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:29.079676 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:29.079877 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:29.079720 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:29.079877 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:29.079786 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:29.079989 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:29.079899 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:29.080053 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:29.080029 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:29.080145 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:29.080108 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:29.705216 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:29.705183 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:29.705953 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:29.705929 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:30.077167 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:30.077146 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 06:58:30.174442 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:30.174364 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" event={"ID":"10e402a1-d12e-4559-aae4-38ca72e7f1cf","Type":"ContainerStarted","Data":"9804e9a5b1232fded06bd3308a25a8baf229957d694268218e1708d9617ef402"} Nov 03 06:58:30.177042 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:30.176942 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v9wll" event={"ID":"580a0715-a63d-43ef-983c-6d459af6ba42","Type":"ContainerStarted","Data":"fcaeb2f338ea1cd0b2c868c00aed52971f5870cac6bfd4b8bc8e88dd24ef4954"} Nov 03 06:58:30.306325 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:30.305322 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-v9wll" podStartSLOduration=2.934343612 podStartE2EDuration="11.305304146s" totalImagesPullingTime="8.370960534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:30.305101862 +0000 UTC m=+11.992993209" watchObservedRunningTime="2026-11-03 06:58:30.305304146 +0000 UTC m=+11.993195517" Nov 03 06:58:30.995993 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:30.995699 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T06:58:30.077164099Z","UUID":"fb91fcbc-43b2-4a70-a0ef-3080ec88ef32","Handler":null,"Name":"","Endpoint":""} Nov 03 06:58:30.997660 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:30.997639 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 06:58:30.997783 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:30.997668 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 06:58:31.077434 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:31.076610 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:31.077434 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:31.076733 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:31.077434 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:31.077109 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:31.077434 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:31.077210 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:31.077434 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:31.077273 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:31.077434 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:31.077345 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:31.180754 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:31.180711 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zq5qq" event={"ID":"eab2a8b5-c6f2-45df-9fcc-161d6ad92498","Type":"ContainerStarted","Data":"feeb2f61a4a16e731425649630b5fce3cc6567b8b92325b65b66ae82824023c8"} Nov 03 06:58:31.183167 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:31.183138 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="4be6b0c47e79f8a4dd66ceb6cd71228392a25291456cecd464ba0e237334201f" exitCode=0 Nov 03 06:58:31.184212 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:31.184145 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerDied","Data":"4be6b0c47e79f8a4dd66ceb6cd71228392a25291456cecd464ba0e237334201f"} Nov 03 06:58:31.426918 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:31.426800 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zq5qq" podStartSLOduration=2.775474036 podStartE2EDuration="11.426780575s" totalImagesPullingTime="8.651306539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:31.300987006 +0000 UTC m=+12.988878353" watchObservedRunningTime="2026-11-03 06:58:31.426780575 +0000 UTC m=+13.114671924" Nov 03 06:58:32.188651 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:32.188576 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" event={"ID":"10e402a1-d12e-4559-aae4-38ca72e7f1cf","Type":"ContainerStarted","Data":"341ffbb5f6ca8cb288cf09c13db060697217d593b827db501cb29a727a552245"} Nov 03 06:58:32.190217 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:32.190183 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vn65v" event={"ID":"c6b39d84-7483-46fb-ab4b-d8de4e3a1212","Type":"ContainerStarted","Data":"32fb82743d813fd35194de0ded0df0ddc79968cc1c8520528f63475c7b5997a4"} Nov 03 06:58:32.312816 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:32.312762 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" podStartSLOduration=3.383289257 podStartE2EDuration="13.312741358s" totalImagesPullingTime="9.929452101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:32.31227152 +0000 UTC m=+14.000162868" watchObservedRunningTime="2026-11-03 06:58:32.312741358 +0000 UTC m=+14.000632706" Nov 03 06:58:32.430874 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:32.430825 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vn65v" podStartSLOduration=3.358817181 podStartE2EDuration="12.43080839s" totalImagesPullingTime="9.071991209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:32.430411725 +0000 UTC m=+14.118303072" watchObservedRunningTime="2026-11-03 06:58:32.43080839 +0000 UTC m=+14.118699738" Nov 03 06:58:33.074635 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:33.074602 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:33.074635 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:33.074618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:33.074860 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:33.074649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:33.074860 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:33.074755 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:33.074860 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:33.074819 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:33.074988 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:33.074888 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:33.194451 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:33.194416 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="2491991c6c6b043dde46c42d8a72829bb31b0559c5f420c663b27cc33405076c" exitCode=0 Nov 03 06:58:33.195005 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:33.194510 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerDied","Data":"2491991c6c6b043dde46c42d8a72829bb31b0559c5f420c663b27cc33405076c"} Nov 03 06:58:33.196080 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:33.196054 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" event={"ID":"95020920-a029-4c17-8ce1-e66a4982a00d","Type":"ContainerStarted","Data":"61c45c20c11bc678895405b044c15eea707283829e566ef256c0c20fa8907380"} Nov 03 06:58:35.074695 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:35.074661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:35.075397 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:35.074812 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:35.075397 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:35.074661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:35.075397 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:35.074897 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:35.075397 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:35.074661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:35.075397 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:35.074971 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:35.203278 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:35.203247 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="149349693e4d9dc9400ce7142dbe637166ca98bac122dd86be259f468c2ea3e1" exitCode=0 Nov 03 06:58:35.203450 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:35.203288 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerDied","Data":"149349693e4d9dc9400ce7142dbe637166ca98bac122dd86be259f468c2ea3e1"} Nov 03 06:58:35.413258 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:35.413165 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" podStartSLOduration=5.584051444 podStartE2EDuration="16.413146547s" totalImagesPullingTime="10.829095103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:33.445955649 +0000 UTC m=+15.133847006" watchObservedRunningTime="2026-11-03 06:58:35.413146547 +0000 UTC m=+17.101037887" Nov 03 06:58:36.906686 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:36.906436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:36.906686 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:36.906665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:36.907175 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:36.906561 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:36.907175 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:36.906793 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:36.907175 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:36.906798 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.90677384 +0000 UTC m=+34.594665182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:36.907175 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:36.906842 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret podName:ca92ce4d-7cf4-4da5-8609-920db260c2d3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.906826236 +0000 UTC m=+34.594717578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret") pod "global-pull-secret-syncer-dtzxb" (UID: "ca92ce4d-7cf4-4da5-8609-920db260c2d3") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:37.008080 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.007847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:37.008232 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:37.008032 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:37.008232 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:37.008126 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:37.008232 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:37.008138 2572 projected.go:196] Error preparing data for projected volume kube-api-access-gc428 for pod openshift-network-diagnostics/network-check-target-dfpc7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:37.008232 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:37.008198 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428 podName:e47cc873-c3fe-421e-8882-9ce4b4ff3737 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:53.008180265 +0000 UTC m=+34.696071605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc428" (UniqueName: "kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428") pod "network-check-target-dfpc7" (UID: "e47cc873-c3fe-421e-8882-9ce4b4ff3737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:37.075143 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.075115 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:37.075303 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.075168 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:37.075303 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.075180 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:37.075303 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:37.075272 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:37.075438 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:37.075327 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:37.075438 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:37.075415 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:37.210826 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.210764 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 06:58:37.211138 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.211116 2572 generic.go:356] "Generic (PLEG): container finished" podID="e6eb01e1-380a-468c-8097-d34540808454" containerID="c3e80a2b6b1c29718af15a534036a799f0a20cdcd185bbe9a38e7ddd1473f129" exitCode=1 Nov 03 06:58:37.211180 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.211165 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"176943a44dc72b695235061eda7cc367d82c2189f5da03b06306724327d462f1"} Nov 03 06:58:37.211228 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.211218 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"4ea6425873328bac9816ea7edef93d590a8166d2d3d0400c96212c10bca00b9e"} Nov 03 06:58:37.211265 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.211253 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"28080ee75e1cf4cca5e639b632a88e5efa26434ba336230eb55232904df2f534"} Nov 03 06:58:37.211302 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.211285 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"46fea575eb10f1cfbc13b5e217ea05803ce37db9ee0154f73b1538f4eda74e5e"} Nov 03 06:58:37.211329 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.211312 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerDied","Data":"c3e80a2b6b1c29718af15a534036a799f0a20cdcd185bbe9a38e7ddd1473f129"} Nov 03 06:58:37.211359 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:37.211340 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"77c2dd38d2fe3c42f8ecfa4e4110418f409e297053b8534ee59125497a7bbda2"} Nov 03 06:58:38.215905 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:38.215870 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="50645334841fad19443b7bb21695330912f95737d20a710a7a0e1e101c0e4b32" exitCode=0 Nov 03 06:58:38.216389 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:38.215929 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerDied","Data":"50645334841fad19443b7bb21695330912f95737d20a710a7a0e1e101c0e4b32"} Nov 03 06:58:39.075701 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:39.075677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:39.075876 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:39.075747 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:39.075876 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:39.075767 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:39.075876 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:39.075856 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:39.076053 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:39.075897 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:39.076053 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:39.075953 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:39.221050 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:39.221000 2572 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="14bd3d450ede31cfcfc3d49dd585d61f13f7b274e1aa20eba5985cfb2327becb" exitCode=0 Nov 03 06:58:39.221537 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:39.221049 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerDied","Data":"14bd3d450ede31cfcfc3d49dd585d61f13f7b274e1aa20eba5985cfb2327becb"} Nov 03 06:58:39.224141 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:39.224125 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 06:58:39.224505 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:39.224482 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"e58f2a1d7cf9df139cb51ae2e25876e4cff1b950eda3772e8ed6bae3dc710b77"} Nov 03 06:58:40.229529 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:40.229497 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" event={"ID":"0dbc159e-0969-417d-91fb-819d526579a3","Type":"ContainerStarted","Data":"2c0083f0b8f20640b6221d9a5b34a8d53223d7cfe6c5b461968443200cf41a4f"} Nov 03 06:58:41.077525 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.077345 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:41.077701 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.077398 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:41.077701 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:41.077611 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:41.077701 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.077405 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:41.077701 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:41.077682 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:41.077841 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:41.077720 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:41.235183 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.235111 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 06:58:41.235804 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.235432 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"4d54463fefa57c7a6db1b2e17ae91e1919d25e1fe8d416b8e411c5d0fdb79b31"} Nov 03 06:58:41.235965 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.235935 2572 scope.go:122] "RemoveContainer" containerID="c3e80a2b6b1c29718af15a534036a799f0a20cdcd185bbe9a38e7ddd1473f129" Nov 03 06:58:41.236131 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.236039 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:41.253685 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.253667 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:41.419484 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:41.419438 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x5cxd" podStartSLOduration=9.784191824 podStartE2EDuration="22.419425752s" totalImagesPullingTime="12.635233928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:40.467334288 +0000 UTC m=+22.155225634" watchObservedRunningTime="2026-11-03 06:58:41.419425752 +0000 UTC m=+23.107317098" Nov 03 06:58:42.241361 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:42.241334 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 06:58:42.241914 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:42.241661 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" event={"ID":"e6eb01e1-380a-468c-8097-d34540808454","Type":"ContainerStarted","Data":"761b877f25a93b68a5797f728c1dfa9120453bed61c90db0347c8b351a19828b"} Nov 03 06:58:42.242052 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:42.242031 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:42.242222 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:42.242207 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:42.257192 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:42.257171 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:58:42.428553 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:42.428511 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" podStartSLOduration=7.622627791 podStartE2EDuration="22.428500024s" totalImagesPullingTime="14.805872233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:42.427423548 +0000 UTC m=+24.115314896" watchObservedRunningTime="2026-11-03 06:58:42.428500024 +0000 UTC m=+24.116391370" Nov 03 06:58:43.074868 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.074829 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:43.074868 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.074858 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:43.075142 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:43.074991 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:43.075142 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.075031 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:43.075142 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:43.075107 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:43.075275 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:43.075154 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:43.273000 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.272967 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:43.273398 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.273106 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 06:58:43.273542 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.273523 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:58:43.597052 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.597003 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dtzxb"] Nov 03 06:58:43.597194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.597115 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:43.597239 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:43.597200 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:43.600157 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.600130 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dfpc7"] Nov 03 06:58:43.600258 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.600216 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:43.600320 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:43.600301 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:43.602570 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.602541 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g8m65"] Nov 03 06:58:43.602666 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:43.602626 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:43.602725 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:43.602711 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:45.077844 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:45.077820 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:45.078180 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:45.077913 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:45.078180 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:45.077925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:45.078180 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:45.077934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:45.078180 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:45.078006 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:45.078180 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:45.078098 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:47.077309 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:47.077279 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:47.077662 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:47.077389 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:47.077662 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:47.077395 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtzxb" podUID="ca92ce4d-7cf4-4da5-8609-920db260c2d3" Nov 03 06:58:47.077662 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:47.077469 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfpc7" podUID="e47cc873-c3fe-421e-8882-9ce4b4ff3737" Nov 03 06:58:47.077662 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:47.077491 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:47.077662 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:47.077545 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8m65" podUID="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" Nov 03 06:58:48.586861 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.586828 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeReady" Nov 03 06:58:48.587227 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.587035 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 06:58:48.703891 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.703813 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hzqt4"] Nov 03 06:58:48.719166 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.719141 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5ndhb"] Nov 03 06:58:48.719286 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.719257 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" Nov 03 06:58:48.731894 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.731739 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4jk57"] Nov 03 06:58:48.731989 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.731846 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" Nov 03 06:58:48.744230 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.744210 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4jk57" Nov 03 06:58:48.754882 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.754860 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54fc5dcf5f-l5q5j"] Nov 03 06:58:48.771565 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.771536 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-299kl"] Nov 03 06:58:48.771696 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.771677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.788436 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht"] Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791761 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqd2\" (UniqueName: \"kubernetes.io/projected/40c1e3ef-149c-401f-8e88-8bdafa0faae6-kube-api-access-blqd2\") pod \"kas-connection-checker-685895c7c9-hzqt4\" (UID: \"40c1e3ef-149c-401f-8e88-8bdafa0faae6\") " pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791801 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-snapshots\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791833 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791859 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791888 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/728637b6-bbb0-4532-adde-4555c4f4153d-serving-cert\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791916 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-service-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791947 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-tmp\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.791991 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.792876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.793088 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.793021 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.804078 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.804048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.804205 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.804099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.804205 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.804133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.806978 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.806945 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.807097 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.806993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpp4t\" (UniqueName: \"kubernetes.io/projected/4e7d1d1e-2fb6-412d-8597-f6f4882d0389-kube-api-access-cpp4t\") pod \"kas-connection-checker-685895c7c9-4jk57\" (UID: \"4e7d1d1e-2fb6-412d-8597-f6f4882d0389\") " pod="kube-system/kas-connection-checker-685895c7c9-4jk57" Nov 03 06:58:48.807097 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.807050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qpj\" (UniqueName: \"kubernetes.io/projected/25276ee9-63e0-4d32-834e-8223c741b724-kube-api-access-b9qpj\") pod \"kas-connection-checker-685895c7c9-5ndhb\" (UID: \"25276ee9-63e0-4d32-834e-8223c741b724\") " pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" Nov 03 06:58:48.807097 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.807081 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrjh6\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.807233 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.807120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qscpq\" (UniqueName: \"kubernetes.io/projected/728637b6-bbb0-4532-adde-4555c4f4153d-kube-api-access-qscpq\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.807527 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.807502 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.807603 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.807584 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.809744 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.809724 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ldktd"] Nov 03 06:58:48.810841 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.810828 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:48.821373 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.821355 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-699757bbb7-drfpj"] Nov 03 06:58:48.821485 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.821468 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:48.829796 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.829775 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57998bb766-8gfrg"] Nov 03 06:58:48.829908 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.829892 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:48.840413 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.840391 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz"] Nov 03 06:58:48.840519 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.840504 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.851225 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.851205 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5"] Nov 03 06:58:48.851351 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.851333 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:48.856373 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.856353 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 06:58:48.856373 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.856366 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 06:58:48.856495 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.856429 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lnjtr\"" Nov 03 06:58:48.856495 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.856469 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 06:58:48.863126 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.863109 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6"] Nov 03 06:58:48.863235 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.863221 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" Nov 03 06:58:48.874654 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.874632 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4"] Nov 03 06:58:48.874762 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.874709 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:48.878493 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.878464 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.878603 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.878546 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 06:58:48.878732 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.878708 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.878849 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.878763 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.879126 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.879104 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.891269 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.891249 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lpk6z"] Nov 03 06:58:48.891395 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.891360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:48.907379 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.907522 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907385 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.907522 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907404 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/66c763d5-1fbd-428a-bf62-cb4102fbc145-kube-api-access-g5gp8\") pod \"volume-data-source-validator-58c9f5f46d-z4nr5\" (UID: \"66c763d5-1fbd-428a-bf62-cb4102fbc145\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" Nov 03 06:58:48.907522 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907427 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:48.907522 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907511 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/271984b8-d201-41a3-a545-690f8a9fad89-serving-cert\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:48.907753 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrjh6\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.907753 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907615 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 06:58:48.907753 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/271984b8-d201-41a3-a545-690f8a9fad89-config\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:48.907753 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2xn\" (UniqueName: \"kubernetes.io/projected/452ab275-c3b0-4c31-8603-7af7cd666346-kube-api-access-2k2xn\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:48.907753 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qscpq\" (UniqueName: \"kubernetes.io/projected/728637b6-bbb0-4532-adde-4555c4f4153d-kube-api-access-qscpq\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.907971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907761 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-trusted-ca\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:48.907971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqd2\" (UniqueName: \"kubernetes.io/projected/40c1e3ef-149c-401f-8e88-8bdafa0faae6-kube-api-access-blqd2\") pod \"kas-connection-checker-685895c7c9-hzqt4\" (UID: \"40c1e3ef-149c-401f-8e88-8bdafa0faae6\") " pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" Nov 03 06:58:48.907971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2vm\" (UniqueName: \"kubernetes.io/projected/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-kube-api-access-cv2vm\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:48.907971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907876 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v"] Nov 03 06:58:48.907971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907946 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:48.907971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.907992 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-default-certificate\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc487\" (UniqueName: \"kubernetes.io/projected/902e5dbc-172c-496e-afaf-2e847ba1f852-kube-api-access-pc487\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:48.908084 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:48.908100 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908106 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:48.908174 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.408134741 +0000 UTC m=+31.096026079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908240 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-service-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908275 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-tmp\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.908298 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpp4t\" (UniqueName: \"kubernetes.io/projected/4e7d1d1e-2fb6-412d-8597-f6f4882d0389-kube-api-access-cpp4t\") pod \"kas-connection-checker-685895c7c9-4jk57\" (UID: \"4e7d1d1e-2fb6-412d-8597-f6f4882d0389\") " pod="kube-system/kas-connection-checker-685895c7c9-4jk57" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qpj\" (UniqueName: \"kubernetes.io/projected/25276ee9-63e0-4d32-834e-8223c741b724-kube-api-access-b9qpj\") pod \"kas-connection-checker-685895c7c9-5ndhb\" (UID: \"25276ee9-63e0-4d32-834e-8223c741b724\") " pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908351 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908383 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908390 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/902e5dbc-172c-496e-afaf-2e847ba1f852-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9wc\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362105ce-9f40-4ad3-964b-472512bf9518-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908805 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-service-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908637 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908879 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxnh\" (UniqueName: \"kubernetes.io/projected/bee72bc7-52f6-45a6-8118-eefba66e6992-kube-api-access-rdxnh\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-config\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.908995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909145 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.909194 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909158 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-tmp\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909179 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2kh4\" (UniqueName: \"kubernetes.io/projected/362105ce-9f40-4ad3-964b-472512bf9518-kube-api-access-n2kh4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362105ce-9f40-4ad3-964b-472512bf9518-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-snapshots\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909359 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909385 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452ab275-c3b0-4c31-8603-7af7cd666346-serving-cert\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-stats-auth\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909438 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909463 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7c5q\" (UniqueName: \"kubernetes.io/projected/271984b8-d201-41a3-a545-690f8a9fad89-kube-api-access-q7c5q\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/728637b6-bbb0-4532-adde-4555c4f4153d-serving-cert\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909626 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:48.909854 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.910392 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.910392 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.909882 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-snapshots\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.912935 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.912911 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.912998 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.912968 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.924999 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.924978 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cpnw7"] Nov 03 06:58:48.925130 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.925088 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" Nov 03 06:58:48.928584 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.928543 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 06:58:48.928766 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.928640 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 06:58:48.928766 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.928658 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.928766 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.928657 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 06:58:48.928766 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.928707 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.930562 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.930540 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.930782 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.930762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.937193 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.937178 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fstk4"] Nov 03 06:58:48.937307 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.937293 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:48.953653 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953638 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hzqt4"] Nov 03 06:58:48.953729 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953666 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4jk57"] Nov 03 06:58:48.953729 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953680 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht"] Nov 03 06:58:48.953729 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953693 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-299kl"] Nov 03 06:58:48.953729 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953707 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54fc5dcf5f-l5q5j"] Nov 03 06:58:48.953729 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953721 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz"] Nov 03 06:58:48.953915 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953734 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5"] Nov 03 06:58:48.953915 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:48.953915 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953747 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57998bb766-8gfrg"] Nov 03 06:58:48.953915 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953880 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lpk6z"] Nov 03 06:58:48.953915 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953908 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4"] Nov 03 06:58:48.954161 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953937 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5ndhb"] Nov 03 06:58:48.954161 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953958 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ldktd"] Nov 03 06:58:48.954161 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.953981 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v"] Nov 03 06:58:48.954161 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.954038 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cpnw7"] Nov 03 06:58:48.954161 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.954110 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-699757bbb7-drfpj"] Nov 03 06:58:48.954161 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.954152 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6"] Nov 03 06:58:48.954337 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.954173 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fstk4"] Nov 03 06:58:48.958433 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958417 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958620 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958644 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958695 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958706 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958715 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958781 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958793 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958837 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958879 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6z6wp\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958904 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958925 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.958979 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 06:58:48.959041 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.959033 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8z7js\"" Nov 03 06:58:48.959653 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.959064 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zhxgw\"" Nov 03 06:58:48.959653 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.959186 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 06:58:48.959653 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.959229 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 06:58:48.959653 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.959234 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xbxhv\"" Nov 03 06:58:48.959653 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.959322 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8gp29\"" Nov 03 06:58:48.959653 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.959398 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9n2tm\"" Nov 03 06:58:48.962051 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.962033 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/728637b6-bbb0-4532-adde-4555c4f4153d-serving-cert\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.976425 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.976485 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.976592 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qscpq\" (UniqueName: \"kubernetes.io/projected/728637b6-bbb0-4532-adde-4555c4f4153d-kube-api-access-qscpq\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.976698 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.976864 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.976899 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.976904 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.977131 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.977138 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.977152 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.977380 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.977478 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.977439 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:48.980648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.977613 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 06:58:48.980648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.980328 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrjh6\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:48.980648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.980485 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqd2\" (UniqueName: \"kubernetes.io/projected/40c1e3ef-149c-401f-8e88-8bdafa0faae6-kube-api-access-blqd2\") pod \"kas-connection-checker-685895c7c9-hzqt4\" (UID: \"40c1e3ef-149c-401f-8e88-8bdafa0faae6\") " pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" Nov 03 06:58:48.980797 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.980649 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpp4t\" (UniqueName: \"kubernetes.io/projected/4e7d1d1e-2fb6-412d-8597-f6f4882d0389-kube-api-access-cpp4t\") pod \"kas-connection-checker-685895c7c9-4jk57\" (UID: \"4e7d1d1e-2fb6-412d-8597-f6f4882d0389\") " pod="kube-system/kas-connection-checker-685895c7c9-4jk57" Nov 03 06:58:48.981532 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:48.981511 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qpj\" (UniqueName: \"kubernetes.io/projected/25276ee9-63e0-4d32-834e-8223c741b724-kube-api-access-b9qpj\") pod \"kas-connection-checker-685895c7c9-5ndhb\" (UID: \"25276ee9-63e0-4d32-834e-8223c741b724\") " pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" Nov 03 06:58:49.010392 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2xn\" (UniqueName: \"kubernetes.io/projected/452ab275-c3b0-4c31-8603-7af7cd666346-kube-api-access-2k2xn\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.010501 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010401 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-trusted-ca\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.010501 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2vm\" (UniqueName: \"kubernetes.io/projected/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-kube-api-access-cv2vm\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.010581 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-default-certificate\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.010618 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010599 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-tmp-dir\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.010648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzkl\" (UniqueName: \"kubernetes.io/projected/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-kube-api-access-pdzkl\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.010682 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc487\" (UniqueName: \"kubernetes.io/projected/902e5dbc-172c-496e-afaf-2e847ba1f852-kube-api-access-pc487\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:49.010682 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010664 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.010773 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.010819 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/902e5dbc-172c-496e-afaf-2e847ba1f852-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:49.010870 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9wc\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.010870 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362105ce-9f40-4ad3-964b-472512bf9518-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:49.010870 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxnh\" (UniqueName: \"kubernetes.io/projected/bee72bc7-52f6-45a6-8118-eefba66e6992-kube-api-access-rdxnh\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:49.011023 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-config\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.011023 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010911 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.011023 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.011023 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010966 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.011023 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.010989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2kh4\" (UniqueName: \"kubernetes.io/projected/362105ce-9f40-4ad3-964b-472512bf9518-kube-api-access-n2kh4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:49.011223 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pw6\" (UniqueName: \"kubernetes.io/projected/936c5f59-3e71-4d99-ad3a-b0762bf8afda-kube-api-access-z2pw6\") pod \"network-check-source-6f78d59f99-fb26v\" (UID: \"936c5f59-3e71-4d99-ad3a-b0762bf8afda\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" Nov 03 06:58:49.011223 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.011223 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362105ce-9f40-4ad3-964b-472512bf9518-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:49.011223 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011173 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.011223 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:49.011351 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011226 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452ab275-c3b0-4c31-8603-7af7cd666346-serving-cert\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.011351 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011252 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-stats-auth\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.011402 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011345 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-trusted-ca\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.011402 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011329 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:49.011459 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011449 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:49.011486 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011466 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:58:49.011486 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7c5q\" (UniqueName: \"kubernetes.io/projected/271984b8-d201-41a3-a545-690f8a9fad89-kube-api-access-q7c5q\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:49.011540 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011501 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch4qf\" (UniqueName: \"kubernetes.io/projected/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-kube-api-access-ch4qf\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:49.011540 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011529 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.511512068 +0000 UTC m=+31.199403410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:58:49.011603 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-config-volume\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.011603 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011585 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-cert\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:49.011655 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011614 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/902e5dbc-172c-496e-afaf-2e847ba1f852-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:49.011655 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011631 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.011655 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011617 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.011655 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011651 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:49.011755 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011711 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.511694265 +0000 UTC m=+31.199585592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:49.011755 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011740 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:58:49.011816 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011788 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.511772861 +0000 UTC m=+31.199664206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:58:49.011880 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.011861 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.51184939 +0000 UTC m=+31.199740749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:58:49.011952 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.012001 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.011976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:49.012072 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012046 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88f82f59-91bd-4845-ae6b-ca143189342d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:49.012072 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012058 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.012188 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.012188 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/66c763d5-1fbd-428a-bf62-cb4102fbc145-kube-api-access-g5gp8\") pod \"volume-data-source-validator-58c9f5f46d-z4nr5\" (UID: \"66c763d5-1fbd-428a-bf62-cb4102fbc145\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" Nov 03 06:58:49.012188 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:49.012188 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/271984b8-d201-41a3-a545-690f8a9fad89-serving-cert\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:49.012391 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/271984b8-d201-41a3-a545-690f8a9fad89-config\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:49.012465 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012444 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362105ce-9f40-4ad3-964b-472512bf9518-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:49.013512 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.012759 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/271984b8-d201-41a3-a545-690f8a9fad89-config\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:49.013512 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.013038 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.013512 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.013123 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:58:49.013512 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.013224 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.513208224 +0000 UTC m=+31.201099551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:58:49.013512 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.013463 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-config\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.013814 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.013507 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:49.014545 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.014520 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452ab275-c3b0-4c31-8603-7af7cd666346-serving-cert\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.014865 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.014847 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362105ce-9f40-4ad3-964b-472512bf9518-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:49.014987 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.014961 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-default-certificate\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.015192 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.015171 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.015815 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.015798 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/271984b8-d201-41a3-a545-690f8a9fad89-serving-cert\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:49.015914 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.015897 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.016083 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.016067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-stats-auth\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.028938 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.028917 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 06:58:49.029073 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.028954 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 06:58:49.029225 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.029209 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kl8t2\"" Nov 03 06:58:49.029285 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.029227 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nd4b5\"" Nov 03 06:58:49.029336 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.029284 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xp6sr\"" Nov 03 06:58:49.029336 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.029293 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 06:58:49.039967 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.039947 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" Nov 03 06:58:49.054587 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.054572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" Nov 03 06:58:49.066910 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.066893 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4jk57" Nov 03 06:58:49.078238 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.078214 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc487\" (UniqueName: \"kubernetes.io/projected/902e5dbc-172c-496e-afaf-2e847ba1f852-kube-api-access-pc487\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:49.078552 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.078530 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 06:58:49.078649 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.078634 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 06:58:49.078782 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.078762 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 06:58:49.078852 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.078810 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 06:58:49.078910 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.078849 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 06:58:49.079127 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.079105 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.079301 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.079279 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9wc\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.079865 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.079847 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxnh\" (UniqueName: \"kubernetes.io/projected/bee72bc7-52f6-45a6-8118-eefba66e6992-kube-api-access-rdxnh\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:49.080245 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.080214 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2vm\" (UniqueName: \"kubernetes.io/projected/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-kube-api-access-cv2vm\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.080304 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.080245 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2xn\" (UniqueName: \"kubernetes.io/projected/452ab275-c3b0-4c31-8603-7af7cd666346-kube-api-access-2k2xn\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.081109 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.081092 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:49.081109 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.081118 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:49.081354 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.081337 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:49.083647 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.083624 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2kh4\" (UniqueName: \"kubernetes.io/projected/362105ce-9f40-4ad3-964b-472512bf9518-kube-api-access-n2kh4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:49.083741 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.083701 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/66c763d5-1fbd-428a-bf62-cb4102fbc145-kube-api-access-g5gp8\") pod \"volume-data-source-validator-58c9f5f46d-z4nr5\" (UID: \"66c763d5-1fbd-428a-bf62-cb4102fbc145\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" Nov 03 06:58:49.084381 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.084361 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7c5q\" (UniqueName: \"kubernetes.io/projected/271984b8-d201-41a3-a545-690f8a9fad89-kube-api-access-q7c5q\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:49.112849 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.112816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.112971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.112895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch4qf\" (UniqueName: \"kubernetes.io/projected/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-kube-api-access-ch4qf\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:49.112971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.112924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-config-volume\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.112971 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.112949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-cert\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:49.113147 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.112984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:49.113147 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.113044 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88f82f59-91bd-4845-ae6b-ca143189342d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:49.113147 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.113115 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-tmp-dir\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.113147 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.113138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzkl\" (UniqueName: \"kubernetes.io/projected/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-kube-api-access-pdzkl\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.113269 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.113232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pw6\" (UniqueName: \"kubernetes.io/projected/936c5f59-3e71-4d99-ad3a-b0762bf8afda-kube-api-access-z2pw6\") pod \"network-check-source-6f78d59f99-fb26v\" (UID: \"936c5f59-3e71-4d99-ad3a-b0762bf8afda\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" Nov 03 06:58:49.113619 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.113602 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:49.113670 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.113660 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.613641251 +0000 UTC m=+31.301532576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:58:49.114383 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.114360 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-config-volume\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.115331 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.115245 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:58:49.115331 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.115266 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-tmp-dir\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.115331 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.115297 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:58:49.615283182 +0000 UTC m=+31.303174522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:58:49.115541 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.115488 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88f82f59-91bd-4845-ae6b-ca143189342d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:49.117099 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.117068 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-cert\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:49.125898 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.125869 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:58:49.136804 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.136739 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 06:58:49.136886 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.136874 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftjg\"" Nov 03 06:58:49.137038 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.136991 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 06:58:49.137114 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.137062 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tq2gc\"" Nov 03 06:58:49.137184 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.137154 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kx4sk\"" Nov 03 06:58:49.139070 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.138999 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:49.171466 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.171311 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:58:49.179229 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.179202 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pw6\" (UniqueName: \"kubernetes.io/projected/936c5f59-3e71-4d99-ad3a-b0762bf8afda-kube-api-access-z2pw6\") pod \"network-check-source-6f78d59f99-fb26v\" (UID: \"936c5f59-3e71-4d99-ad3a-b0762bf8afda\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" Nov 03 06:58:49.180532 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.180511 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch4qf\" (UniqueName: \"kubernetes.io/projected/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-kube-api-access-ch4qf\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:49.180618 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.180541 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzkl\" (UniqueName: \"kubernetes.io/projected/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-kube-api-access-pdzkl\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.187879 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.187651 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" Nov 03 06:58:49.208345 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.208280 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:58:49.264841 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.262911 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" Nov 03 06:58:49.291381 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.291331 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:58:49.417337 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.417133 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:49.417443 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.417284 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:49.417443 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.417400 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:58:49.417526 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.417453 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.417437412 +0000 UTC m=+32.105328739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:58:49.480626 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.478041 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5ndhb"] Nov 03 06:58:49.482965 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.482938 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4jk57"] Nov 03 06:58:49.483085 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.483028 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hzqt4"] Nov 03 06:58:49.493841 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.493760 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7d1d1e_2fb6_412d_8597_f6f4882d0389.slice/crio-d0bab689ae469a9472f79b5138483c30f959a1b35da74ca9bdc6d18de76afd62 WatchSource:0}: Error finding container d0bab689ae469a9472f79b5138483c30f959a1b35da74ca9bdc6d18de76afd62: Status 404 returned error can't find the container with id d0bab689ae469a9472f79b5138483c30f959a1b35da74ca9bdc6d18de76afd62 Nov 03 06:58:49.520443 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.520408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.520550 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.520463 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:49.520550 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.520493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:49.520671 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.520552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:49.520671 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.520647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:49.520920 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.520901 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:58:49.520976 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.520966 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.520949024 +0000 UTC m=+32.208840371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:58:49.521075 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521060 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:58:49.521125 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521105 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.521093626 +0000 UTC m=+32.208984953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:58:49.521179 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521173 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.521163882 +0000 UTC m=+32.209055207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:58:49.521250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521236 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:49.521298 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521254 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:58:49.521298 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521283 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.521273927 +0000 UTC m=+32.209165269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:58:49.521396 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521336 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:49.521396 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.521366 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.521356907 +0000 UTC m=+32.209248232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:49.559403 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.556820 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-299kl"] Nov 03 06:58:49.560068 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.560026 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728637b6_bbb0_4532_adde_4555c4f4153d.slice/crio-6a85974966c3320c1b9e54f55de9b962f03ea4bd3e1226d0f7460175954450b1 WatchSource:0}: Error finding container 6a85974966c3320c1b9e54f55de9b962f03ea4bd3e1226d0f7460175954450b1: Status 404 returned error can't find the container with id 6a85974966c3320c1b9e54f55de9b962f03ea4bd3e1226d0f7460175954450b1 Nov 03 06:58:49.583264 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.583238 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ldktd"] Nov 03 06:58:49.591428 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.591393 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452ab275_c3b0_4c31_8603_7af7cd666346.slice/crio-4a55e5db4901f63d26e494ef11712aaea6601c91952fc9b2fb24c9e3b654d7eb WatchSource:0}: Error finding container 4a55e5db4901f63d26e494ef11712aaea6601c91952fc9b2fb24c9e3b654d7eb: Status 404 returned error can't find the container with id 4a55e5db4901f63d26e494ef11712aaea6601c91952fc9b2fb24c9e3b654d7eb Nov 03 06:58:49.593133 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.593034 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz"] Nov 03 06:58:49.595436 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.595418 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362105ce_9f40_4ad3_964b_472512bf9518.slice/crio-be120e3bc42ee7cc3d92d23d606b1a8efd91d9419b20723b7e363c746a968866 WatchSource:0}: Error finding container be120e3bc42ee7cc3d92d23d606b1a8efd91d9419b20723b7e363c746a968866: Status 404 returned error can't find the container with id be120e3bc42ee7cc3d92d23d606b1a8efd91d9419b20723b7e363c746a968866 Nov 03 06:58:49.610090 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.610067 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5"] Nov 03 06:58:49.614981 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.614960 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c763d5_1fbd_428a_bf62_cb4102fbc145.slice/crio-987d14d3adfd8f0b73aaa703cad765c77ca9e175a470537eab0d080109b8b3c7 WatchSource:0}: Error finding container 987d14d3adfd8f0b73aaa703cad765c77ca9e175a470537eab0d080109b8b3c7: Status 404 returned error can't find the container with id 987d14d3adfd8f0b73aaa703cad765c77ca9e175a470537eab0d080109b8b3c7 Nov 03 06:58:49.621660 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.621639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:49.621735 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.621689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:49.621800 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.621782 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:49.621831 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.621803 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:58:49.621868 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.621839 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.621825312 +0000 UTC m=+32.309716637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:58:49.621868 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:49.621858 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:58:50.621846016 +0000 UTC m=+32.309737341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:58:49.629924 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.629907 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6"] Nov 03 06:58:49.632083 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.632063 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271984b8_d201_41a3_a545_690f8a9fad89.slice/crio-bbb42e5d565467256e3b0034017f7c9f3147d935d2df14181923e6a3b5b44740 WatchSource:0}: Error finding container bbb42e5d565467256e3b0034017f7c9f3147d935d2df14181923e6a3b5b44740: Status 404 returned error can't find the container with id bbb42e5d565467256e3b0034017f7c9f3147d935d2df14181923e6a3b5b44740 Nov 03 06:58:49.674555 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.674530 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v"] Nov 03 06:58:49.677363 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.677338 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936c5f59_3e71_4d99_ad3a_b0762bf8afda.slice/crio-1bd4c6b277ce6a88f217ba423c12216c443bd02c05c706e8490736ffc4f3f94a WatchSource:0}: Error finding container 1bd4c6b277ce6a88f217ba423c12216c443bd02c05c706e8490736ffc4f3f94a: Status 404 returned error can't find the container with id 1bd4c6b277ce6a88f217ba423c12216c443bd02c05c706e8490736ffc4f3f94a Nov 03 06:58:49.694149 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.694127 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fstk4"] Nov 03 06:58:49.696644 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:49.696619 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8346ef93_89cc_47f7_b3bf_3a309b6efa2d.slice/crio-928086259a3ecfdcfff6c01e88f9428da99c82a99a4842f0661f74cc3e7b0730 WatchSource:0}: Error finding container 928086259a3ecfdcfff6c01e88f9428da99c82a99a4842f0661f74cc3e7b0730: Status 404 returned error can't find the container with id 928086259a3ecfdcfff6c01e88f9428da99c82a99a4842f0661f74cc3e7b0730 Nov 03 06:58:49.984516 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:49.984479 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:50.247568 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.247486 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:50.267391 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.267355 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" event={"ID":"66c763d5-1fbd-428a-bf62-cb4102fbc145","Type":"ContainerStarted","Data":"987d14d3adfd8f0b73aaa703cad765c77ca9e175a470537eab0d080109b8b3c7"} Nov 03 06:58:50.268469 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.268448 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-299kl" event={"ID":"728637b6-bbb0-4532-adde-4555c4f4153d","Type":"ContainerStarted","Data":"6a85974966c3320c1b9e54f55de9b962f03ea4bd3e1226d0f7460175954450b1"} Nov 03 06:58:50.269485 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.269457 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fstk4" event={"ID":"8346ef93-89cc-47f7-b3bf-3a309b6efa2d","Type":"ContainerStarted","Data":"928086259a3ecfdcfff6c01e88f9428da99c82a99a4842f0661f74cc3e7b0730"} Nov 03 06:58:50.270753 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.270733 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4jk57" event={"ID":"4e7d1d1e-2fb6-412d-8597-f6f4882d0389","Type":"ContainerStarted","Data":"5c7ba5c8714f1981fdcc5e99141be2aad0237c33eaa5b84fb27b396707137eac"} Nov 03 06:58:50.270828 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.270766 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4jk57" event={"ID":"4e7d1d1e-2fb6-412d-8597-f6f4882d0389","Type":"ContainerStarted","Data":"d0bab689ae469a9472f79b5138483c30f959a1b35da74ca9bdc6d18de76afd62"} Nov 03 06:58:50.271838 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.271823 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" event={"ID":"362105ce-9f40-4ad3-964b-472512bf9518","Type":"ContainerStarted","Data":"be120e3bc42ee7cc3d92d23d606b1a8efd91d9419b20723b7e363c746a968866"} Nov 03 06:58:50.272851 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.272831 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" event={"ID":"452ab275-c3b0-4c31-8603-7af7cd666346","Type":"ContainerStarted","Data":"4a55e5db4901f63d26e494ef11712aaea6601c91952fc9b2fb24c9e3b654d7eb"} Nov 03 06:58:50.274131 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.274112 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" event={"ID":"25276ee9-63e0-4d32-834e-8223c741b724","Type":"ContainerStarted","Data":"1c5837954f9fec18d450d814bf0f69e619f5258778c353d9f52f409ed5dd4e35"} Nov 03 06:58:50.274213 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.274141 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" event={"ID":"25276ee9-63e0-4d32-834e-8223c741b724","Type":"ContainerStarted","Data":"52b04e9ef70e2f615a74f3f1a1a528699906a0790663c99bd57e768cfd694756"} Nov 03 06:58:50.275232 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.275214 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" event={"ID":"271984b8-d201-41a3-a545-690f8a9fad89","Type":"ContainerStarted","Data":"bbb42e5d565467256e3b0034017f7c9f3147d935d2df14181923e6a3b5b44740"} Nov 03 06:58:50.276471 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.276444 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" event={"ID":"40c1e3ef-149c-401f-8e88-8bdafa0faae6","Type":"ContainerStarted","Data":"9bb7533443647a2eb5e4fa06595fb5712725f65fa25e1bb97d8399f7e60b5542"} Nov 03 06:58:50.276566 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.276483 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" event={"ID":"40c1e3ef-149c-401f-8e88-8bdafa0faae6","Type":"ContainerStarted","Data":"d5b091b0bb1fd312f026eeab76a861d8004a713217b6747b52119a40868c8021"} Nov 03 06:58:50.277522 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.277494 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" event={"ID":"936c5f59-3e71-4d99-ad3a-b0762bf8afda","Type":"ContainerStarted","Data":"1bd4c6b277ce6a88f217ba423c12216c443bd02c05c706e8490736ffc4f3f94a"} Nov 03 06:58:50.429157 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.429119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:50.429298 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.429274 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:50.429298 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.429292 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:58:50.429408 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.429357 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.429336026 +0000 UTC m=+34.117227367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:58:50.529799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.529722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:50.529799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.529767 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:50.529799 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.529797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.529856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.529878 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.529891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.529935 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.529913614 +0000 UTC m=+34.217804940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.529956 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.529960 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.529978 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.529999 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.529962 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.529952358 +0000 UTC m=+34.217843684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.530048 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.530032954 +0000 UTC m=+34.217924290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:50.530058 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.530061 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.530054797 +0000 UTC m=+34.217946122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:58:50.530400 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.530078 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.530072585 +0000 UTC m=+34.217963910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:58:50.630735 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.630704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:50.631247 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:50.630791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:50.631247 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.630943 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:58:50.631247 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.631001 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.630983473 +0000 UTC m=+34.318874813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:58:50.631247 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.631134 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:50.631247 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:50.631177 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:52.631163842 +0000 UTC m=+34.319055169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:58:51.013749 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:51.013339 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4jk57" podStartSLOduration=327.013288356 podStartE2EDuration="5m27.013288356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:51.011462279 +0000 UTC m=+32.699353627" watchObservedRunningTime="2026-11-03 06:58:51.013288356 +0000 UTC m=+32.701179701" Nov 03 06:58:51.133567 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:51.133514 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" podStartSLOduration=327.133499991 podStartE2EDuration="5m27.133499991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:51.132394365 +0000 UTC m=+32.820285713" watchObservedRunningTime="2026-11-03 06:58:51.133499991 +0000 UTC m=+32.821391375" Nov 03 06:58:51.192691 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:51.192649 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:51.251672 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:51.251619 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" podStartSLOduration=327.251602127 podStartE2EDuration="5m27.251602127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:51.250657535 +0000 UTC m=+32.938548886" watchObservedRunningTime="2026-11-03 06:58:51.251602127 +0000 UTC m=+32.939493474" Nov 03 06:58:51.400340 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:51.400222 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:52.450260 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.449485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:52.450260 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.449717 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:52.450260 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.449736 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:58:52.450260 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.449792 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.449774411 +0000 UTC m=+38.137665759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.550961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.551034 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.551065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.551127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.551157 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.551406 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.551466 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.551449344 +0000 UTC m=+38.239340678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.551847 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.551891 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.551877382 +0000 UTC m=+38.239768714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.551955 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.55194434 +0000 UTC m=+38.239835672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.552032 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.552043 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.552072 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.552062477 +0000 UTC m=+38.239953807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.552121 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:52.552250 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.552150 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.552141145 +0000 UTC m=+38.240032471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:52.652275 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.652235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:52.652445 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.652418 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:58:52.652622 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.652510 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.6524898 +0000 UTC m=+38.340381128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:58:52.652622 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.652513 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:52.652622 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.652421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:52.652622 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.652600 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.652590085 +0000 UTC m=+38.340481456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:58:52.956960 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.955972 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:58:52.956960 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.956164 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:52.956960 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.956874 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:58:52.956960 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:52.956932 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:24.956915778 +0000 UTC m=+66.644807103 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : secret "metrics-daemon-secret" not found Nov 03 06:58:52.963969 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:52.963879 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:53.018688 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.018426 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:58:53.057841 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.057294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:53.077685 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.074451 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:53.294855 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.294771 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" event={"ID":"66c763d5-1fbd-428a-bf62-cb4102fbc145","Type":"ContainerStarted","Data":"1269d0b0a2741654e96303719239f1e7caf9b3ea47cc3e809da09b603d321e6c"} Nov 03 06:58:53.367375 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.367336 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:53.416920 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.416865 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" podStartSLOduration=295.862798568 podStartE2EDuration="4m58.416845948s" totalImagesPullingTime="2.55404738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:53.415361496 +0000 UTC m=+35.103252846" watchObservedRunningTime="2026-11-03 06:58:53.416845948 +0000 UTC m=+35.104737296" Nov 03 06:58:53.436701 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:53.435596 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca92ce4d_7cf4_4da5_8609_920db260c2d3.slice/crio-390e6e4d5813fba641cec64092070688ea0900a0945203470dc4feb07f307c0c WatchSource:0}: Error finding container 390e6e4d5813fba641cec64092070688ea0900a0945203470dc4feb07f307c0c: Status 404 returned error can't find the container with id 390e6e4d5813fba641cec64092070688ea0900a0945203470dc4feb07f307c0c Nov 03 06:58:53.439269 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.437729 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dtzxb"] Nov 03 06:58:53.612156 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.612121 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:53.831378 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.831338 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:53.855310 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:53.855242 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dfpc7"] Nov 03 06:58:53.861051 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:53.860972 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47cc873_c3fe_421e_8882_9ce4b4ff3737.slice/crio-8348e72ade41ce52bc4fa9a930f066b0bb4bb3e76c0d2834954d080086cf26ce WatchSource:0}: Error finding container 8348e72ade41ce52bc4fa9a930f066b0bb4bb3e76c0d2834954d080086cf26ce: Status 404 returned error can't find the container with id 8348e72ade41ce52bc4fa9a930f066b0bb4bb3e76c0d2834954d080086cf26ce Nov 03 06:58:54.300462 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:54.300339 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dfpc7" event={"ID":"e47cc873-c3fe-421e-8882-9ce4b4ff3737","Type":"ContainerStarted","Data":"8348e72ade41ce52bc4fa9a930f066b0bb4bb3e76c0d2834954d080086cf26ce"} Nov 03 06:58:54.302569 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:54.302513 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dtzxb" event={"ID":"ca92ce4d-7cf4-4da5-8609-920db260c2d3","Type":"ContainerStarted","Data":"390e6e4d5813fba641cec64092070688ea0900a0945203470dc4feb07f307c0c"} Nov 03 06:58:55.132777 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:55.132734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:55.329208 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:55.329173 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:56.312068 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.311966 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" event={"ID":"271984b8-d201-41a3-a545-690f8a9fad89","Type":"ContainerStarted","Data":"a359b80fe77b0170ae2238d20d89b0862f04ba267047e2e813e15c3a8c17b22e"} Nov 03 06:58:56.314760 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.314725 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" event={"ID":"936c5f59-3e71-4d99-ad3a-b0762bf8afda","Type":"ContainerStarted","Data":"d034cd33cbffa1cfa2cedde192842f36f3558e489790dfb717a1be2c456e5bd1"} Nov 03 06:58:56.317238 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.316849 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-299kl" event={"ID":"728637b6-bbb0-4532-adde-4555c4f4153d","Type":"ContainerStarted","Data":"f08c6194fe153a992f77cf467b10d686a1f673c5b4c4f091e149bf9022f09f54"} Nov 03 06:58:56.318608 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.318586 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fstk4" event={"ID":"8346ef93-89cc-47f7-b3bf-3a309b6efa2d","Type":"ContainerStarted","Data":"a0acfc0589aa0a125483134e9a2e976a643e6a671c111c9b26bd15b3c9f96a5c"} Nov 03 06:58:56.320520 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.320145 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dfpc7" event={"ID":"e47cc873-c3fe-421e-8882-9ce4b4ff3737","Type":"ContainerStarted","Data":"7ac6f6332de06bdb0f4f2efc66acae98f708be54a1ac40277095065503f5a2ac"} Nov 03 06:58:56.320520 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.320207 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:58:56.323316 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.323294 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" event={"ID":"362105ce-9f40-4ad3-964b-472512bf9518","Type":"ContainerStarted","Data":"ef67d3a9f2e404168899a7121231c2f013d90ac7dd44c7bd3e11218dc598e2e9"} Nov 03 06:58:56.325613 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.325593 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/0.log" Nov 03 06:58:56.325730 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.325629 2572 generic.go:356] "Generic (PLEG): container finished" podID="452ab275-c3b0-4c31-8603-7af7cd666346" containerID="0f9c7f3c47e788c3ada02794139cccc466821c6830ad0bdede79d03c71b44e52" exitCode=255 Nov 03 06:58:56.325730 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.325649 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" event={"ID":"452ab275-c3b0-4c31-8603-7af7cd666346","Type":"ContainerDied","Data":"0f9c7f3c47e788c3ada02794139cccc466821c6830ad0bdede79d03c71b44e52"} Nov 03 06:58:56.325851 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.325822 2572 scope.go:122] "RemoveContainer" containerID="0f9c7f3c47e788c3ada02794139cccc466821c6830ad0bdede79d03c71b44e52" Nov 03 06:58:56.434950 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.434876 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" podStartSLOduration=295.450946743 podStartE2EDuration="5m1.434857975s" totalImagesPullingTime="5.983911232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:56.433608627 +0000 UTC m=+38.121499974" watchObservedRunningTime="2026-11-03 06:58:56.434857975 +0000 UTC m=+38.122749327" Nov 03 06:58:56.498221 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.498092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:58:56.498439 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.498412 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:56.498439 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.498433 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:58:56.498555 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.498498 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.498480364 +0000 UTC m=+46.186371709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.599475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.599513 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.599532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.599580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.599612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599635 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599656 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.599636628 +0000 UTC m=+46.287527988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599711 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.599693938 +0000 UTC m=+46.287585268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599730 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599734 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599742 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599767 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599771 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.599759117 +0000 UTC m=+46.287650460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:58:56.599779 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599787 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.599779992 +0000 UTC m=+46.287671317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:58:56.600379 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.599864 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.599848768 +0000 UTC m=+46.287740108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:58:56.678539 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.678485 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fstk4" podStartSLOduration=2.758716594 podStartE2EDuration="8.678471274s" totalImagesPullingTime="5.91975468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:56.677568065 +0000 UTC m=+38.365459412" watchObservedRunningTime="2026-11-03 06:58:56.678471274 +0000 UTC m=+38.366362621" Nov 03 06:58:56.678999 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.678970 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" podStartSLOduration=295.659019545 podStartE2EDuration="5m1.678962124s" totalImagesPullingTime="6.019942579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:56.556823302 +0000 UTC m=+38.244714650" watchObservedRunningTime="2026-11-03 06:58:56.678962124 +0000 UTC m=+38.366853464" Nov 03 06:58:56.700505 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.700452 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:58:56.700987 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.700675 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:58:56.700987 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.700863 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:58:56.700987 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.700907 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.700895623 +0000 UTC m=+46.388786947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:58:56.700987 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.700949 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:56.700987 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:56.700969 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:59:04.700963187 +0000 UTC m=+46.388854512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:58:56.800279 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.800229 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-299kl" podStartSLOduration=295.747637422 podStartE2EDuration="5m1.800213663s" totalImagesPullingTime="6.052576241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:56.799713699 +0000 UTC m=+38.487605048" watchObservedRunningTime="2026-11-03 06:58:56.800213663 +0000 UTC m=+38.488105011" Nov 03 06:58:56.922762 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:56.922656 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" podStartSLOduration=295.956425265 podStartE2EDuration="5m1.922640378s" totalImagesPullingTime="5.966215113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:56.922179194 +0000 UTC m=+38.610070542" watchObservedRunningTime="2026-11-03 06:58:56.922640378 +0000 UTC m=+38.610531759" Nov 03 06:58:57.051094 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.051056 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:57.168830 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.168765 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dfpc7" podStartSLOduration=36.124961996 podStartE2EDuration="38.168743971s" totalImagesPullingTime="2.043781975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:57.167180486 +0000 UTC m=+38.855071834" watchObservedRunningTime="2026-11-03 06:58:57.168743971 +0000 UTC m=+38.856635317" Nov 03 06:58:57.260321 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.260242 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:57.313806 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.313772 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd"] Nov 03 06:58:57.332938 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.332916 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/1.log" Nov 03 06:58:57.333486 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.333469 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/0.log" Nov 03 06:58:57.333586 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.333504 2572 generic.go:356] "Generic (PLEG): container finished" podID="452ab275-c3b0-4c31-8603-7af7cd666346" containerID="ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3" exitCode=255 Nov 03 06:58:57.334728 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.334707 2572 scope.go:122] "RemoveContainer" containerID="ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3" Nov 03 06:58:57.334920 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:57.334890 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ldktd_openshift-console-operator(452ab275-c3b0-4c31-8603-7af7cd666346)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" podUID="452ab275-c3b0-4c31-8603-7af7cd666346" Nov 03 06:58:57.335606 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.335582 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" event={"ID":"452ab275-c3b0-4c31-8603-7af7cd666346","Type":"ContainerDied","Data":"ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3"} Nov 03 06:58:57.335698 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.335619 2572 scope.go:122] "RemoveContainer" containerID="0f9c7f3c47e788c3ada02794139cccc466821c6830ad0bdede79d03c71b44e52" Nov 03 06:58:57.335698 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.335684 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" Nov 03 06:58:57.380673 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.380640 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jjq57"] Nov 03 06:58:57.399527 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.399504 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:57.399648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.399511 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p8ttv\"" Nov 03 06:58:57.399648 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.399581 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 06:58:57.409983 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.409957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhbg\" (UniqueName: \"kubernetes.io/projected/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233-kube-api-access-gnhbg\") pod \"migrator-66df5d496b-29ktd\" (UID: \"1bfb1d47-8ee3-4e09-a4ea-ddcef2244233\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" Nov 03 06:58:57.412895 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.412875 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd"] Nov 03 06:58:57.412993 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.412976 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.445391 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.445368 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jjq57"] Nov 03 06:58:57.463514 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.463489 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:57.499425 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.499398 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 06:58:57.499425 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.499404 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 06:58:57.499425 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.499427 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dznd\"" Nov 03 06:58:57.511852 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.511790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.511852 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.511837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnfg9\" (UniqueName: \"kubernetes.io/projected/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-api-access-tnfg9\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.512059 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.512035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhbg\" (UniqueName: \"kubernetes.io/projected/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233-kube-api-access-gnhbg\") pod \"migrator-66df5d496b-29ktd\" (UID: \"1bfb1d47-8ee3-4e09-a4ea-ddcef2244233\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" Nov 03 06:58:57.512099 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.512088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/018a989a-6542-4db1-bd2a-d80df7bbed24-crio-socket\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.512171 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.512156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.512217 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.512196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/018a989a-6542-4db1-bd2a-d80df7bbed24-data-volume\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.580987 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.580955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhbg\" (UniqueName: \"kubernetes.io/projected/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233-kube-api-access-gnhbg\") pod \"migrator-66df5d496b-29ktd\" (UID: \"1bfb1d47-8ee3-4e09-a4ea-ddcef2244233\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" Nov 03 06:58:57.612814 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.612783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/018a989a-6542-4db1-bd2a-d80df7bbed24-crio-socket\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.612972 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.612838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.612972 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.612871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/018a989a-6542-4db1-bd2a-d80df7bbed24-data-volume\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.612972 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.612907 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/018a989a-6542-4db1-bd2a-d80df7bbed24-crio-socket\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.613178 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:57.612913 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:58:57.613178 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.612992 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.613178 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.613049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfg9\" (UniqueName: \"kubernetes.io/projected/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-api-access-tnfg9\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.613178 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:57.613069 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.113050326 +0000 UTC m=+39.800941653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : configmap "kube-rbac-proxy" not found Nov 03 06:58:57.613178 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:57.613082 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:58:57.613178 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:57.613132 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.113121814 +0000 UTC m=+39.801013139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:58:57.613433 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.613210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/018a989a-6542-4db1-bd2a-d80df7bbed24-data-volume\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.665481 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.665448 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" Nov 03 06:58:57.683649 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.683625 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfg9\" (UniqueName: \"kubernetes.io/projected/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-api-access-tnfg9\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:57.710920 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.710886 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:57.944430 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:57.944400 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:58.059148 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.059120 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd"] Nov 03 06:58:58.061474 ip-10-0-136-184 kubenswrapper[2572]: W1103 06:58:58.061442 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfb1d47_8ee3_4e09_a4ea_ddcef2244233.slice/crio-a9f66f322326a0231ca2730a13f0c66879e0be70906ab0fe0f3d95fcea759c39 WatchSource:0}: Error finding container a9f66f322326a0231ca2730a13f0c66879e0be70906ab0fe0f3d95fcea759c39: Status 404 returned error can't find the container with id a9f66f322326a0231ca2730a13f0c66879e0be70906ab0fe0f3d95fcea759c39 Nov 03 06:58:58.117825 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.117786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:58.117969 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.117890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:58.117969 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:58.117954 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:58:58.118100 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:58.118045 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.118028819 +0000 UTC m=+40.805920161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:58:58.118100 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:58.118054 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:58:58.118212 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:58.118113 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.118096882 +0000 UTC m=+40.805988222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : configmap "kube-rbac-proxy" not found Nov 03 06:58:58.177026 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.176975 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:58.339149 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.339108 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" event={"ID":"1bfb1d47-8ee3-4e09-a4ea-ddcef2244233","Type":"ContainerStarted","Data":"a9f66f322326a0231ca2730a13f0c66879e0be70906ab0fe0f3d95fcea759c39"} Nov 03 06:58:58.340906 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.340878 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/1.log" Nov 03 06:58:58.341291 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.341273 2572 scope.go:122] "RemoveContainer" containerID="ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3" Nov 03 06:58:58.341520 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:58.341489 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ldktd_openshift-console-operator(452ab275-c3b0-4c31-8603-7af7cd666346)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" podUID="452ab275-c3b0-4c31-8603-7af7cd666346" Nov 03 06:58:58.499379 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.499332 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:58.727793 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:58.727694 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:59.128147 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:59.128113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:59.128325 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:59.128263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:58:59.128441 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:59.128425 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:58:59.128508 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:59.128489 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.128472014 +0000 UTC m=+42.816363342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:58:59.128736 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:59.128721 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:58:59.128810 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:59.128771 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.12875752 +0000 UTC m=+42.816648848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : configmap "kube-rbac-proxy" not found Nov 03 06:58:59.139607 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:59.139582 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:59.139724 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:59.139640 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:58:59.197658 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:59.197634 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:58:59.346234 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:59.346209 2572 scope.go:122] "RemoveContainer" containerID="ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3" Nov 03 06:58:59.346714 ip-10-0-136-184 kubenswrapper[2572]: E1103 06:58:59.346422 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ldktd_openshift-console-operator(452ab275-c3b0-4c31-8603-7af7cd666346)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" podUID="452ab275-c3b0-4c31-8603-7af7cd666346" Nov 03 06:58:59.957851 ip-10-0-136-184 kubenswrapper[2572]: I1103 06:58:59.957718 2572 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:58:59.957880 ip-10-0-136-184 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 06:58:59.978403 ip-10-0-136-184 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 06:58:59.978568 ip-10-0-136-184 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 06:58:59.978728 ip-10-0-136-184 systemd[1]: kubelet.service: Consumed 3.018s CPU time. Nov 03 06:58:59.993706 ip-10-0-136-184 systemd[1]: Starting Kubernetes Kubelet... Nov 03 06:59:00.096369 ip-10-0-136-184 kubenswrapper[4925]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:59:00.096369 ip-10-0-136-184 kubenswrapper[4925]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 06:59:00.096369 ip-10-0-136-184 kubenswrapper[4925]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:59:00.096369 ip-10-0-136-184 kubenswrapper[4925]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:59:00.098683 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098599 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.098683 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098683 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.098683 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098687 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098691 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098694 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098697 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098700 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098702 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098705 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098707 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098710 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098713 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098715 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098718 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098721 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098723 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098726 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098729 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098737 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098739 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098742 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098744 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.098779 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098748 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098750 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098753 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098755 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098758 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098761 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098764 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098766 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098769 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098771 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098775 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098778 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098781 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098783 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098786 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098788 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098791 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098794 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098797 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098799 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.099288 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098801 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098804 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098806 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098809 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098812 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098815 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098819 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098823 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098827 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098830 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098833 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098835 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098838 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098841 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098844 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098846 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098851 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098854 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098856 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.099763 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098859 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098862 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098865 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098868 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098871 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098875 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098879 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098882 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098885 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098887 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098890 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098893 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098895 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098898 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098901 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098903 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098906 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098909 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098911 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.100245 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098914 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098917 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098919 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098922 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098924 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098927 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098929 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098932 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098934 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098937 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098941 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098943 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098946 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098949 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098952 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098955 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098957 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098960 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098962 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098965 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.100719 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098967 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098970 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098973 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098975 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098978 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.098981 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099795 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099804 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099807 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099810 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099813 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099816 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099819 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099822 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099824 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099827 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099830 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099833 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099835 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099838 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.101225 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099841 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099844 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099848 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099851 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099853 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099856 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099859 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099861 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099863 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099866 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099869 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099871 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099874 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099876 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099879 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099882 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099884 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099887 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099890 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.101741 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099892 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099895 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099898 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099900 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099904 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099908 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099911 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099914 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099918 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099921 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099923 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099926 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099928 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099931 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099933 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099937 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099940 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099942 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099945 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.102211 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099947 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099949 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099952 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099954 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099957 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099959 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099962 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099964 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099966 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099969 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099971 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099974 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099976 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099979 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099981 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099984 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099986 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099990 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099992 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099995 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.102685 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.099999 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100002 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100004 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100026 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100029 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100031 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100034 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100036 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100041 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100043 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100046 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100048 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100051 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100054 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100056 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100059 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100061 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100064 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100066 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100069 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.103195 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100071 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100074 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100076 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100079 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100081 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100084 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100088 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100091 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100094 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100097 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100100 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100102 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100107 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100109 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100670 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100677 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100680 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100682 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100685 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.103663 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100688 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100692 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100695 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100698 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100700 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100703 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100705 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100708 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100711 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100713 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100716 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100719 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100721 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100724 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100727 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100729 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100733 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100735 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100738 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100740 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.104163 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100743 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100745 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100748 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100751 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100753 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100758 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100760 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100764 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100766 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100769 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100771 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100774 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100776 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100779 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100782 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100785 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100788 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100790 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100792 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100795 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.104634 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100797 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100800 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100802 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100805 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100807 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100809 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100812 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100815 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100818 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100822 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100825 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100828 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100830 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100833 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100835 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100838 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100840 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100844 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100847 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100850 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.105169 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100853 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100855 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100858 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100860 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100863 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100866 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100868 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100871 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100873 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100876 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100878 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100881 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100883 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100886 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100888 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100891 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100893 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100896 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100898 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.105656 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100901 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100903 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100905 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100908 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100911 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100914 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100917 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100920 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100923 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100926 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100930 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100933 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100936 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100940 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100944 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100947 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100950 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100952 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100955 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.106185 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100957 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100960 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.100962 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101072 4925 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101092 4925 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101099 4925 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101103 4925 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101107 4925 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101110 4925 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101115 4925 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101119 4925 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101122 4925 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101125 4925 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101128 4925 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101133 4925 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101136 4925 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101139 4925 flags.go:64] FLAG: --cgroup-root="" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101142 4925 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101145 4925 flags.go:64] FLAG: --client-ca-file="" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101147 4925 flags.go:64] FLAG: --cloud-provider="external" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101150 4925 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101157 4925 flags.go:64] FLAG: --cluster-domain="" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101159 4925 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 06:59:00.106645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101164 4925 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101167 4925 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101170 4925 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101175 4925 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101178 4925 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101186 4925 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101190 4925 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101193 4925 flags.go:64] FLAG: --contention-profiling="false" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101196 4925 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101198 4925 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101202 4925 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101204 4925 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101209 4925 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101212 4925 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101214 4925 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101219 4925 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101222 4925 flags.go:64] FLAG: --enable-server="true" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101224 4925 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101232 4925 flags.go:64] FLAG: --event-burst="100" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101235 4925 flags.go:64] FLAG: --event-qps="50" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101238 4925 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101242 4925 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101245 4925 flags.go:64] FLAG: --eviction-hard="" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101249 4925 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 06:59:00.107228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101252 4925 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101255 4925 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101258 4925 flags.go:64] FLAG: --eviction-soft="" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101261 4925 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101264 4925 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101266 4925 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101269 4925 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101272 4925 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101276 4925 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101278 4925 flags.go:64] FLAG: --feature-gates="" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101282 4925 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101285 4925 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101289 4925 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101292 4925 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101307 4925 flags.go:64] FLAG: --healthz-port="10248" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101310 4925 flags.go:64] FLAG: --help="false" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101313 4925 flags.go:64] FLAG: --hostname-override="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101316 4925 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101319 4925 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101322 4925 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101325 4925 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101329 4925 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101332 4925 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 06:59:00.107810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101335 4925 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101338 4925 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101341 4925 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101344 4925 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101347 4925 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101350 4925 flags.go:64] FLAG: --kube-reserved="" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101352 4925 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101355 4925 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101358 4925 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101361 4925 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101364 4925 flags.go:64] FLAG: --lock-file="" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101367 4925 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101369 4925 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101372 4925 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101383 4925 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101386 4925 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101389 4925 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101391 4925 flags.go:64] FLAG: --logging-format="text" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101394 4925 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101397 4925 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101400 4925 flags.go:64] FLAG: --manifest-url="" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101403 4925 flags.go:64] FLAG: --manifest-url-header="" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101407 4925 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101410 4925 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101419 4925 flags.go:64] FLAG: --max-pods="110" Nov 03 06:59:00.108420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101423 4925 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101425 4925 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101428 4925 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101431 4925 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101434 4925 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101437 4925 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101440 4925 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101447 4925 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101450 4925 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101453 4925 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101456 4925 flags.go:64] FLAG: --pod-cidr="" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101459 4925 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101462 4925 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101465 4925 flags.go:64] FLAG: --pods-per-core="0" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101468 4925 flags.go:64] FLAG: --port="10250" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101471 4925 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101474 4925 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0adb07709ae12134a" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101477 4925 flags.go:64] FLAG: --qos-reserved="" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101480 4925 flags.go:64] FLAG: --read-only-port="10255" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101483 4925 flags.go:64] FLAG: --register-node="true" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101486 4925 flags.go:64] FLAG: --register-with-taints="" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101489 4925 flags.go:64] FLAG: --registry-burst="10" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101492 4925 flags.go:64] FLAG: --registry-qps="5" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101495 4925 flags.go:64] FLAG: --reserved-cpus="" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101497 4925 flags.go:64] FLAG: --reserved-memory="" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101501 4925 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 06:59:00.109110 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101504 4925 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101507 4925 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101510 4925 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101513 4925 flags.go:64] FLAG: --runonce="false" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101516 4925 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101519 4925 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101522 4925 flags.go:64] FLAG: --seccomp-default="false" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101525 4925 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101528 4925 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101531 4925 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101534 4925 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101537 4925 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101540 4925 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101544 4925 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101547 4925 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101550 4925 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101553 4925 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101556 4925 flags.go:64] FLAG: --system-cgroups="" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101559 4925 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101564 4925 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101567 4925 flags.go:64] FLAG: --tls-cert-file="" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101570 4925 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101577 4925 flags.go:64] FLAG: --tls-min-version="" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101580 4925 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101583 4925 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 06:59:00.109758 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101585 4925 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101588 4925 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101591 4925 flags.go:64] FLAG: --v="2" Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101596 4925 flags.go:64] FLAG: --version="false" Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101600 4925 flags.go:64] FLAG: --vmodule="" Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101603 4925 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.101606 4925 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101747 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101752 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101755 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101758 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101761 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101764 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101766 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101769 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101772 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101774 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101777 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101779 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101781 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101786 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.110398 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101789 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101791 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101794 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101798 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101800 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101803 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101805 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101808 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101810 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101813 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101815 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101818 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101820 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101823 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101825 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101828 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101830 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101833 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101835 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101838 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.111111 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101840 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101843 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101845 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101848 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101851 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101853 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101856 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101859 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101861 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101864 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101867 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101873 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101876 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101878 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101881 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101885 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101887 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101889 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101892 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.111898 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101895 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101897 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101900 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101902 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101905 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101908 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101910 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101913 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101915 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101918 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101920 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101923 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101925 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101927 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101930 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101932 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101935 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101937 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101940 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.112639 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101942 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101945 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101948 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101950 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101952 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101956 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101959 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101961 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101964 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101969 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101971 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101974 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101976 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101979 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101981 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101984 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101986 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101988 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101991 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101994 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.113317 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101996 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.101999 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102001 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102004 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102019 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102022 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102025 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102028 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102031 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102033 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102036 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102038 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102041 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102043 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.102070 4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102112 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.113853 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102117 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102121 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102123 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102126 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102129 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102131 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102135 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102138 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102140 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102143 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102145 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102148 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102150 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102153 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102155 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102158 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102160 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102163 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102165 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102168 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.114270 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102170 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102173 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102175 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102178 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102180 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102184 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102188 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102191 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102194 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102197 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102199 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102202 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102205 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102215 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102217 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102220 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102222 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102225 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102229 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.114935 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102232 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102234 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102237 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102239 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102242 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102245 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102248 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102250 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102253 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102255 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102258 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102260 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102263 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102265 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102268 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102270 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102273 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102275 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102278 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102280 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.115468 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102283 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102285 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102288 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102290 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102293 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102302 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102306 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102308 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102311 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102313 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102317 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102322 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102325 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102328 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102330 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102333 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102335 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102338 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102341 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.116081 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102343 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102349 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102351 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102354 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102356 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102359 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102361 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102364 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102366 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102369 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102371 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102373 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102376 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102378 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102381 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102383 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102385 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102388 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102390 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.116538 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102400 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102402 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102405 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102408 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102410 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102414 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102417 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.102419 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.106244 4925 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.106260 4925 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.106278 4925 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.106283 4925 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106338 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106342 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106346 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106349 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106352 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106356 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106359 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106361 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.117020 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106364 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106367 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106369 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106372 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106374 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106377 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106380 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106382 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106385 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106387 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106390 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106392 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106395 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106398 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106401 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106403 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106406 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106410 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106413 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106416 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.117514 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106418 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106422 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106425 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106427 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106430 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106432 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106435 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106438 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106440 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106442 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106445 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106447 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106450 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106452 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106455 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106458 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106460 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106462 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106465 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.118005 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106469 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106472 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106475 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106478 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106481 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106483 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106486 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106489 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106492 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106494 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106497 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106500 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106502 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106505 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106507 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106510 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106513 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106515 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106518 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106520 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.118483 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106523 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106525 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106528 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106530 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106533 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106535 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106538 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106540 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106543 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106545 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106547 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106550 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106553 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106555 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106558 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106561 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106565 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106568 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106571 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.118959 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106574 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106576 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106580 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106582 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106585 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106587 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106590 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106592 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106595 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106598 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106600 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106603 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106605 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106608 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106610 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106613 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106615 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106618 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106620 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.119505 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106622 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.106649 4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106785 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106789 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106792 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106795 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106797 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106800 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106803 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106805 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106808 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106810 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106813 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106815 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106818 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106820 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.119965 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106823 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106825 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106828 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106831 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106833 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106835 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106838 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106841 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106843 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106846 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106848 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106851 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106853 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106856 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106858 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106860 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106863 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106865 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106868 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106870 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106873 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.120363 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106875 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106877 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106880 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106882 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106885 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106887 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106890 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106892 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106895 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106897 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106900 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106902 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106905 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106907 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106910 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106912 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106914 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106917 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106919 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106922 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.120903 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106925 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106927 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106929 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106932 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106934 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106937 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106939 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106941 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106944 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106947 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106949 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106952 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106954 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106956 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106959 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106963 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106966 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106970 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106973 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.121405 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106976 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106979 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106982 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106984 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106987 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106989 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106992 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106994 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106997 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.106999 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107001 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107004 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107023 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107029 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107033 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107036 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107039 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107041 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107044 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.121923 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107046 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107049 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107053 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107056 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107059 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107062 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107064 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107067 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107070 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107076 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107079 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107082 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107085 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.107108 4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107141 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:00.122412 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107145 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107149 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107152 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107154 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107157 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107159 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107162 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107166 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107170 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107174 4925 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107177 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107180 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107187 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107190 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107193 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107196 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107198 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107200 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107203 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:00.122777 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107205 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107208 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107211 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107213 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107216 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107218 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107221 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107224 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107228 4925 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107231 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107233 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107236 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107239 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107244 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107248 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107251 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107253 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107256 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107258 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:00.123266 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107261 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107264 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107266 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107269 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107271 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107274 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107276 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107280 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107282 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107286 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107288 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107291 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107293 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107295 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107298 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107300 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107303 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107305 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107308 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107311 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:00.123764 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107313 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107317 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107320 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107323 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107325 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107328 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107330 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107333 4925 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107335 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107338 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107340 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107343 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107345 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107348 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107350 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107352 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107355 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107357 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107360 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107362 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107365 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:00.124384 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107368 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107370 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107373 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107375 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107377 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107380 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107382 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107385 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107387 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107390 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107392 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107395 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107399 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107401 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107404 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107406 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107409 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107411 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107414 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107416 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:00.124915 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107419 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107422 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107425 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107427 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107430 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:00.107434 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.107488 4925 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.108670 4925 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.108731 4925 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.108913 4925 server.go:1014] "Starting client certificate rotation" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.109035 4925 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.109082 4925 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 06:53:19 +0000 UTC" deadline="2028-07-31 06:44:19.00198639 +0000 UTC" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.109109 4925 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15263h45m18.892879158s" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.109354 4925 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.110410 4925 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.112930 4925 log.go:25] "Validated CRI v1 runtime API" Nov 03 06:59:00.125462 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.114615 4925 log.go:25] "Validated CRI v1 image API" Nov 03 06:59:00.125866 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.115461 4925 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 06:59:00.125866 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.117343 4925 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80006ed6-ff77-4f95-873f-129682c0524f:/dev/nvme0n1p3 a61abdef-5501-49b6-a028-ba81258537d9:/dev/nvme0n1p4] Nov 03 06:59:00.125940 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.117365 4925 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bd4c6b277ce6a88f217ba423c12216c443bd02c05c706e8490736ffc4f3f94a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bd4c6b277ce6a88f217ba423c12216c443bd02c05c706e8490736ffc4f3f94a/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30fd3627b9c4432586acd0a7ebc07375d416be91d21f2fab256d2a4ad5c62f91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30fd3627b9c4432586acd0a7ebc07375d416be91d21f2fab256d2a4ad5c62f91/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/390e6e4d5813fba641cec64092070688ea0900a0945203470dc4feb07f307c0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/390e6e4d5813fba641cec64092070688ea0900a0945203470dc4feb07f307c0c/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d06ad07fb20cba3df1f1a5d88db3399ffa319a36a76708a1a30268825e90267/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d06ad07fb20cba3df1f1a5d88db3399ffa319a36a76708a1a30268825e90267/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a55e5db4901f63d26e494ef11712aaea6601c91952fc9b2fb24c9e3b654d7eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a55e5db4901f63d26e494ef11712aaea6601c91952fc9b2fb24c9e3b654d7eb/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52b04e9ef70e2f615a74f3f1a1a528699906a0790663c99bd57e768cfd694756/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52b04e9ef70e2f615a74f3f1a1a528699906a0790663c99bd57e768cfd694756/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bc1bf10904b637729a1dec0e044c79a0e3050e11d3d3a258df22ef86e5cac59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bc1bf10904b637729a1dec0e044c79a0e3050e11d3d3a258df22ef86e5cac59/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63abe62800a3852c2c711306dc0100fb3397a7ba6cf9d0280f3bfd0943d1c591/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63abe62800a3852c2c711306dc0100fb3397a7ba6cf9d0280f3bfd0943d1c591/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a85974966c3320c1b9e54f55de9b962f03ea4bd3e1226d0f7460175954450b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a85974966c3320c1b9e54f55de9b962f03ea4bd3e1226d0f7460175954450b1/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/724b5a7fe3285e73476c8ae0592a9cec7b229b3af70609c0e384cd3bc420c87c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/724b5a7fe3285e73476c8ae0592a9cec7b229b3af70609c0e384cd3bc420c87c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78c3063a3ea8935541cf2ba5c4bbb74c0e17ff83679aae44b1a2c200e577721d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78c3063a3ea8935541cf2ba5c4bbb74c0e17ff83679aae44b1a2c200e577721d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8348e72ade41ce52bc4fa9a930f066b0bb4bb3e76c0d2834954d080086cf26ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8348e72ade41ce52bc4fa9a930f066b0bb4bb3e76c0d2834954d080086cf26ce/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9193dc5965f783ebccd05cf612d68f63fc4044a31db970b5a26e18cdf78cfb7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9193dc5965f783ebccd05cf612d68f63fc4044a31db970b5a26e18cdf78cfb7c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/928086259a3ecfdcfff6c01e88f9428da99c82a99a4842f0661f74cc3e7b0730/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/928086259a3ecfdcfff6c01e88f9428da99c82a99a4842f0661f74cc3e7b0730/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/987d14d3adfd8f0b73aaa703cad765c77ca9e175a470537eab0d080109b8b3c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/987d14d3adfd8f0b73aaa703cad765c77ca9e175a470537eab0d080109b8b3c7/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9f66f322326a0231ca2730a13f0c66879e0be70906ab0fe0f3d95fcea759c39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9f66f322326a0231ca2730a13f0c66879e0be70906ab0fe0f3d95fcea759c39/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbb42e5d565467256e3b0034017f7c9f3147d935d2df14181923e6a3b5b44740/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbb42e5d565467256e3b0034017f7c9f3147d935d2df14181923e6a3b5b44740/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be120e3bc42ee7cc3d92d23d606b1a8efd91d9419b20723b7e363c746a968866/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be120e3bc42ee7cc3d92d23d606b1a8efd91d9419b20723b7e363c746a968866/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0bab689ae469a9472f79b5138483c30f959a1b35da74ca9bdc6d18de76afd62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0bab689ae469a9472f79b5138483c30f959a1b35da74ca9bdc6d18de76afd62/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5b091b0bb1fd312f026eeab76a861d8004a713217b6747b52119a40868c8021/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5b091b0bb1fd312f026eeab76a861d8004a713217b6747b52119a40868c8021/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d94e5adca559e4458bbef324a1d9b3f18209be7f3c4142fc14adb36715755231/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d94e5adca559e4458bbef324a1d9b3f18209be7f3c4142fc14adb36715755231/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/018a989a-6542-4db1-bd2a-d80df7bbed24/volumes/kubernetes.io~projected/kube-api-access-tnfg9:{mountpoint:/var/lib/kubelet/pods/018a989a-6542-4db1-bd2a-d80df7bbed24/volumes/kubernetes.io~projected/kube-api-access-tnfg9 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~projected/kube-api-access-gk9wc:{mountpoint:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~projected/kube-api-access-gk9wc major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dbc159e-0969-417d-91fb-819d526579a3/volumes/kubernetes.io~projected/kube-api-access-qtjv9:{mountpoint:/var/lib/kubelet/pods/0dbc159e-0969-417d-91fb-819d526579a3/volumes/kubernetes.io~projected/kube-api-access-qtjv9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10e402a1-d12e-4559-aae4-38ca72e7f1cf/volumes/kubernetes.io~projected/kube-api-access-ckqdw:{mountpoint:/var/lib/kubelet/pods/10e402a1-d12e-4559-aae4-38ca72e7f1cf/volumes/kubernetes.io~projected/kube-api-access-ckqdw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233/volumes/kubernetes.io~projected/kube-api-access-gnhbg:{mountpoint:/var/lib/kubelet/pods/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233/volumes/kubernetes.io~projected/kube-api-access-gnhbg major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25276ee9-63e0-4d32-834e-8223c741b724/volumes/kubernetes.io~projected/kube-api-access-b9qpj:{mountpoint:/var/lib/kubelet/pods/25276ee9-63e0-4d32-834e-8223c741b724/volumes/kubernetes.io~projected/kube-api-access-b9qpj major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/271984b8-d201-41a3-a545-690f8a9fad89/volumes/kubernetes.io~projected/kube-api-access-q7c5q:{mountpoint:/var/lib/kubelet/pods/271984b8-d201-41a3-a545-690f8a9fad89/volumes/kubernetes.io~projected/kube-api-access-q7c5q major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/271984b8-d201-41a3-a545-690f8a9fad89/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/271984b8-d201-41a3-a545-690f8a9fad89/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fc2a19a-80f8-4318-a6fd-b246b3022d0e/volumes/kubernetes.io~projected/kube-api-access-sffvz:{mountpoint:/var/lib/kubelet/pods/2fc2a19a-80f8-4318-a6fd-b246b3022d0e/volumes/kubernetes.io~projected/kube-api-access-sffvz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/362105ce-9f40-4ad3-964b-472512bf9518/volumes/kubernetes.io~projected/kube-api-access-n2kh4:{mountpoint:/var/lib/kubelet/pods/362105ce-9f40-4ad3-964b-472512bf9518/volumes/kubernetes.io~projected/kube-api-access-n2kh4 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/362105ce-9f40-4ad3-964b-472512bf9518/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/362105ce-9f40-4ad3-964b-472512bf9518/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c1e3ef-149c-401f-8e88-8bdafa0faae6/volumes/kubernetes.io~projected/kube-api-access-blqd2:{mountpoint:/var/lib/kubelet/pods/40c1e3ef-149c-401f-8e88-8bdafa0faae6/volumes/kubernetes.io~projected/kube-api-access-blqd2 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/452ab275-c3b0-4c31-8603-7af7cd666346/volumes/kubernetes.io~projected/kube-api-access-2k2xn:{mountpoint:/var/lib/kubelet/pods/452ab275-c3b0-4c31-8603-7af7cd666346/volumes/kubernetes.io~projected/kube-api-access-2k2xn major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/452ab275-c3b0-4c31-8603-7af7cd666346/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/452ab275-c3b0-4c31-8603-7af7cd666346/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e7d1d1e-2fb6-412d-8597-f6f4882d0389/volumes/kubernetes.io~projected/kube-api-access-cpp4t:{mountpoint:/var/lib/kubelet/pods/4e7d1d1e-2fb6-412d-8597-f6f4882d0389/volumes/kubernetes.io~projected/kube-api-access-cpp4t major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/580a0715-a63d-43ef-983c-6d459af6ba42/volumes/kubernetes.io~projected/kube-api-access-wxfgw:{mountpoint:/var/lib/kubelet/pods/580a0715-a63d-43ef-983c-6d459af6ba42/volumes/kubernetes.io~projected/kube-api-access-wxfgw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66c763d5-1fbd-428a-bf62-cb4102fbc145/volumes/kubernetes.io~projected/kube-api-access-g5gp8:{mountpoint:/var/lib/kubelet/pods/66c763d5-1fbd-428a-bf62-cb4102fbc145/volumes/kubernetes.io~projected/kube-api-access-g5gp8 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/728637b6-bbb0-4532-adde-4555c4f4153d/volumes/kubernetes.io~projected/kube-api-access-qscpq:{mountpoint:/var/lib/kubelet/pods/728637b6-bbb0-4532-adde-4555c4f4153d/volumes/kubernetes.io~projected/kube-api-access-qscpq major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/728637b6-bbb0-4532-adde-4555c4f4153d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/728637b6-bbb0-4532-adde-4555c4f4153d/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8346ef93-89cc-47f7-b3bf-3a309b6efa2d/volumes/kubernetes.io~projected/kube-api-access-ch4qf:{mountpoint:/var/lib/kubelet/pods/8346ef93-89cc-47f7-b3bf-3a309b6efa2d/volumes/kubernetes.io~projected/kube-api-access-ch4qf major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8346ef93-89cc-47f7-b3bf-3a309b6efa2d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8346ef93-89cc-47f7-b3bf-3a309b6efa2d/volumes/kubernetes.io~secret/cert major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~projected/kube-api-access-cv2vm:{mountpoint:/var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~projected/kube-api-access-cv2vm major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~secret/default-certificate major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~secret/stats-auth major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902e5dbc-172c-496e-afaf-2e847ba1f852/volumes/kubernetes.io~projected/kube-api-access-pc487:{mountpoint:/var/lib/kubelet/pods/902e5dbc-172c-496e-afaf-2e847ba1f852/volumes/kubernetes.io~projected/kube-api-access-pc487 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/936c5f59-3e71-4d99-ad3a-b0762bf8afda/volumes/kubernetes.io~projected/kube-api-access-z2pw6:{mountpoint:/var/lib/kubelet/pods/936c5f59-3e71-4d99-ad3a-b0762bf8afda/volumes/kubernetes.io~projected/kube-api-access-z2pw6 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~projected/kube-api-access-p4f7r:{mountpoint:/var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~projected/kube-api-access-p4f7r major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bee72bc7-52f6-45a6-8118-eefba66e6992/volumes/kubernetes.io~projected/kube-api-access-rdxnh:{mountpoint:/var/lib/kubelet/pods/bee72bc7-52f6-45a6-8118-eefba66e6992/volumes/kubernetes.io~projected/kube-api-access-rdxnh major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6b39d84-7483-46fb-ab4b-d8de4e3a1212/volumes/kubernetes.io~projected/kube-api-access-bh76m:{mountpoint:/var/lib/kubelet/pods/c6b39d84-7483-46fb-ab4b-d8de4e3a1212/volumes/kubernetes.io~projected/kube-api-access-bh76m major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3/volumes/kubernetes.io~projected/kube-api-access-cj2xf:{mountpoint:/var/lib/kubelet/pods/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3/volumes/kubernetes.io~projected/kube-api-access-cj2xf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca92ce4d-7cf4-4da5-8609-920db260c2d3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ca92ce4d-7cf4-4da5-8609-920db260c2d3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c/volumes/kubernetes.io~projected/kube-api-access-pdzkl:{mountpoint:/var/lib/kubelet/pods/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c/volumes/kubernetes.io~projected/kube-api-access-pdzkl major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e47cc873-c3fe-421e-8882-9ce4b4ff3737/volumes/kubernetes.io~projected/kube-api-access-gc428:{mountpoint:/var/lib/kubelet/pods/e47cc873-c3fe-421e-8882-9ce4b4ff3737/volumes/kubernetes.io~projected/kube-api-access-gc428 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volumes/kubernetes.io~projected/kube-api-access-tc9m9:{mountpoint:/var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volumes/kubernetes.io~projected/kube-api-access-tc9m9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eab2a8b5-c6f2-45df-9fcc-161d6ad92498/volumes/kubernetes.io~projected/kube-api-access-2glrf:{mountpoint:/var/lib/kubelet/pods/eab2a8b5-c6f2-45df-9fcc-161d6ad92498/volumes/kubernetes.io~projected/kube-api-access-2glrf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~projected/kube-api-access-rrjh6:{mountpoint:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~projected/kube-api-access-rrjh6 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7781680-4a88-4a3f-8fc1-61c702b89f63/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f7781680-4a88-4a3f-8fc1-61c702b89f63/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e377754771749427af1f0d168fcb1acbc626d9a911cdd90ab9cab8f98660b7bd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4b8d1c2525b2bfa9c47f5482f4f47625a3c8e8b34bae945649a0ddfde217e5b4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3d12a686f87e453d879ee8c4c248bf84068f423855decb48514b2271be18733a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/6920a4d5404e4dc2502af6070ed84a6a81447b75d6f0c35a14abe4359be71025/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c5c4cca63ffae907d04e382cc8695a0275ef8999a60a2f6cef93df62af39865f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0bd6f5d61afd3e2e333c5575d962058741644ef4fa29a415f103d8b908bea4a2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ae49f1e149b57e856ba0dfe332e31779f9a07f9d5a3cfe18bac707ba229fb879/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/673f5fb0473dfac3a9b449528482e1c485cbf05ffca7dccbb247733dd1972a90/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e57d69c7b2ce63067ebe95e273008ae1346fc6d061bed4ddbbf033b513812efa/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c4a03322cb1d6ffeffdc0786f197e4c75c21113e892d83a85707aa4288a8c6a3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7e475236bdcb9fd0c2225261cf184cd8b964be6f3715a1c4f0749431e5b7a2fe/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/2639c8b5a0d6063d0fe911d995e591b5dd1ec15613676ab59b3a2b426df31720/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a2e6efd7ca30f91b73a31d524c5c3c85760f8b4c847b2781fbb5ce7219a11cf8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/95bc3a847e3927cd8a38f41b9881fe1587827f160e30f0272ce2dd62084ddc97/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6acd398aef89cecd9e0191efc4c92cb3841b25b2406a7ad5c7849d3e25aa6f4b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c198b4c08421ca1fbe3be166d852db72be4dad3307ab2f9634c04e80af5ed460/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/aae05480785d36940326b6616b127d6639d6bba806370eb0d8b6f4aed3968da2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/42ef356e56a0dc02a374e7de8bcccb548f308ae888dc8e76ed2e6934ab776769/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dd5b3f78b9dfdb6dd7ed4bf5557df857f993c736aff3c73b3e997acd9a55d0fc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/59aadfa28435f8504ea228e119f106125ffd068c812c3e2b1a5d58b5285fd3f0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1ca319f9928242130de568b71243e71968a4f063f7a6c26f27018621fb2e63a2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4356a5fc25df8b5d23e40c2f1cd2338e9f13322f2bef56884a3db9206d4ef2a5/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/ba8f7078ad7387ef610e9aff8b87d153bd13b334ded21a5329dea56bf643612e/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/e7997e101564a2ec2c34aed95139ac8f8d890928f6d1209b8ec5d664ad4aa9a7/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/2f0b801aa89fc708db499e6545b29807548d79ca411e5e55bfcf6c8789fc9370/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b1794af54fe18d1d75bbd957f9ceae91cf9604b7a79a6f6bb11ae46f45c35cd1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8992ec58c5b70410d36b73ae12af417dd741ce3d930997426cf1cb416718093f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/55dbea4f131b345dc02d1cf7e46850fbb129d39fe4185e6de63e0b5c17e3a1fa/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/38c027ef3cf1683dee863ee2a8ceb70f0e09c8a2e665d4a7859cebd141a58538/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/52a128d1c743928603c65245c4c38d2aac08582942e5fc92a4ae8e8a475574ea/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/894e19c8622621c6a33b35348ac2b50a838371017d3fc723ea33c42d1fdc0614/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/1086ac012a01c35b8e001569d96d27c43050dff29c29ccf24f3814a426ce4ade/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/47b2d18cf5c4db5a0c9bfe53eaae5a9e1eeea9fad537b8fda855d5dbcf7c5f88/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/3dca8872086365010f1bc6f576603112564c1018cace048fb4ad6dff6f4e892d/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/d09100be13d754357f35467f1f92a606e020602d4251f780f35526e17af810b1/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/57536cf1b11988818cfc591aaa5dd8127574ae5180b04d1a5a22132b6c1a7a81/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/64e283138197724019d81ad208fb4d5483a20a8a74b8420dfdcde04507f43c49/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/9927dda7a3393019087b26b1a0589cb02a2c2acdee4c1716d8736a56a9beec31/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/08397c4092380e202db34e531ff2bd8c0fb1b01082a682d14595725308ae5a64/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b94b3432773a653491b7b36341e72283bb2b25c94fc2b203d1ee7095177a442b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/5c32b4ebf1b75e3f21b63e108cda5efa62afcea7a25a9f14eda6e141e8da283f/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/88df5826e085fb0998f4559578307f5d3bf543bc97585fc3215c1c73e777e88e/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/e42e06f1a89d253727b61620157d8f45174a8f7424e983ea5a4cf869e6f0b83d/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/257de49d75db80f9e8d43e884c4b474d0ad264d3ae0c4e271bc4586910682001/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/a9448a59527f769296a45a2b9f67785b164e778e2c30aad92f37f1392f6d964e/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/69f36655f27354d053aa25faca0020537127977a8784448ccd2e0b4ffdeb58be/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/304d0ef7f5a4f61550e85782432ae40d5762d3d9185c33ef2840b29ae33df71d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/78cbbd616d9907399812bc57d04dde57dc50b49845b9ee275d965beff67ed9e0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/6366baeaf370f76b42c0165df4e723d008252022fe77bbbc64c3a13406e7794f/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/11958f1622967d4d3233a2121397292c8cdabee3c0f978cafd4faf7bdf754a38/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/df4592eeee961b72a671e26e201d5d3b67bf56e9e36a05bba8def9b9116cbe8b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a117055e4e616ee7bbfeb7c67a5f2a27b365cbf04c7f7ef5d329cbce8f98c58f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0def0eadc26c53044626dddfdef2185a1186d10a8b814e0999fc801d26cedbc9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b83f77cec2853d6ebb4766e5e94f904daa13210a13ccb4956355f5acb4dff10a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/47e13ca5f9ee89f958b783f812761c033f52ba580adb6febf593b4662a20c8cd/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.125173 4925 manager.go:217] Machine: {Timestamp:2026-11-03 06:59:00.124683518 +0000 UTC m=+0.058212171 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100424 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27927827b97665a72aeeb56de8f254 SystemUUID:ec279278-27b9-7665-a72a-eeb56de8f254 BootID:e8f9580c-58ab-4870-b942-2b040345e9d7 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0dbc159e-0969-417d-91fb-819d526579a3/volumes/kubernetes.io~projected/kube-api-access-qtjv9 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25276ee9-63e0-4d32-834e-8223c741b724/volumes/kubernetes.io~projected/kube-api-access-b9qpj DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52b04e9ef70e2f615a74f3f1a1a528699906a0790663c99bd57e768cfd694756/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volumes/kubernetes.io~projected/kube-api-access-tc9m9 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5b091b0bb1fd312f026eeab76a861d8004a713217b6747b52119a40868c8021/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eab2a8b5-c6f2-45df-9fcc-161d6ad92498/volumes/kubernetes.io~projected/kube-api-access-2glrf DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d06ad07fb20cba3df1f1a5d88db3399ffa319a36a76708a1a30268825e90267/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78c3063a3ea8935541cf2ba5c4bbb74c0e17ff83679aae44b1a2c200e577721d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/724b5a7fe3285e73476c8ae0592a9cec7b229b3af70609c0e384cd3bc420c87c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bd4c6b277ce6a88f217ba423c12216c443bd02c05c706e8490736ffc4f3f94a/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6eb01e1-380a-468c-8097-d34540808454/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/271984b8-d201-41a3-a545-690f8a9fad89/volumes/kubernetes.io~projected/kube-api-access-q7c5q DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902e5dbc-172c-496e-afaf-2e847ba1f852/volumes/kubernetes.io~projected/kube-api-access-pc487 DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a85974966c3320c1b9e54f55de9b962f03ea4bd3e1226d0f7460175954450b1/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d94e5adca559e4458bbef324a1d9b3f18209be7f3c4142fc14adb36715755231/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3/volumes/kubernetes.io~projected/kube-api-access-cj2xf DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bc1bf10904b637729a1dec0e044c79a0e3050e11d3d3a258df22ef86e5cac59/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbb42e5d565467256e3b0034017f7c9f3147d935d2df14181923e6a3b5b44740/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2fc2a19a-80f8-4318-a6fd-b246b3022d0e/volumes/kubernetes.io~projected/kube-api-access-sffvz DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~projected/kube-api-access-cv2vm DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~projected/kube-api-access-p4f7r DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/728637b6-bbb0-4532-adde-4555c4f4153d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c/volumes/kubernetes.io~projected/kube-api-access-pdzkl DeviceMajor:0 DeviceMinor:237 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/018a989a-6542-4db1-bd2a-d80df7bbed24/volumes/kubernetes.io~projected/kube-api-access-tnfg9 DeviceMajor:0 DeviceMinor:319 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/10e402a1-d12e-4559-aae4-38ca72e7f1cf/volumes/kubernetes.io~projected/kube-api-access-ckqdw DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9193dc5965f783ebccd05cf612d68f63fc4044a31db970b5a26e18cdf78cfb7c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40c1e3ef-149c-401f-8e88-8bdafa0faae6/volumes/kubernetes.io~projected/kube-api-access-blqd2 DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bee72bc7-52f6-45a6-8118-eefba66e6992/volumes/kubernetes.io~projected/kube-api-access-rdxnh DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be120e3bc42ee7cc3d92d23d606b1a8efd91d9419b20723b7e363c746a968866/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ca92ce4d-7cf4-4da5-8609-920db260c2d3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30fd3627b9c4432586acd0a7ebc07375d416be91d21f2fab256d2a4ad5c62f91/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/452ab275-c3b0-4c31-8603-7af7cd666346/volumes/kubernetes.io~projected/kube-api-access-2k2xn DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/362105ce-9f40-4ad3-964b-472512bf9518/volumes/kubernetes.io~projected/kube-api-access-n2kh4 DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63abe62800a3852c2c711306dc0100fb3397a7ba6cf9d0280f3bfd0943d1c591/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/271984b8-d201-41a3-a545-690f8a9fad89/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0bab689ae469a9472f79b5138483c30f959a1b35da74ca9bdc6d18de76afd62/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e47cc873-c3fe-421e-8882-9ce4b4ff3737/volumes/kubernetes.io~projected/kube-api-access-gc428 DeviceMajor:0 DeviceMinor:295 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/452ab275-c3b0-4c31-8603-7af7cd666346/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8346ef93-89cc-47f7-b3bf-3a309b6efa2d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8346ef93-89cc-47f7-b3bf-3a309b6efa2d/volumes/kubernetes.io~projected/kube-api-access-ch4qf DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/987d14d3adfd8f0b73aaa703cad765c77ca9e175a470537eab0d080109b8b3c7/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233/volumes/kubernetes.io~projected/kube-api-access-gnhbg DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~projected/kube-api-access-gk9wc DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/390e6e4d5813fba641cec64092070688ea0900a0945203470dc4feb07f307c0c/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9f66f322326a0231ca2730a13f0c66879e0be70906ab0fe0f3d95fcea759c39/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f7781680-4a88-4a3f-8fc1-61c702b89f63/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/936c5f59-3e71-4d99-ad3a-b0762bf8afda/volumes/kubernetes.io~projected/kube-api-access-z2pw6 DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8348e72ade41ce52bc4fa9a930f066b0bb4bb3e76c0d2834954d080086cf26ce/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a55e5db4901f63d26e494ef11712aaea6601c91952fc9b2fb24c9e3b654d7eb/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/580a0715-a63d-43ef-983c-6d459af6ba42/volumes/kubernetes.io~projected/kube-api-access-wxfgw DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/728637b6-bbb0-4532-adde-4555c4f4153d/volumes/kubernetes.io~projected/kube-api-access-qscpq DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e7d1d1e-2fb6-412d-8597-f6f4882d0389/volumes/kubernetes.io~projected/kube-api-access-cpp4t DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/928086259a3ecfdcfff6c01e88f9428da99c82a99a4842f0661f74cc3e7b0730/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c6b39d84-7483-46fb-ab4b-d8de4e3a1212/volumes/kubernetes.io~projected/kube-api-access-bh76m DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66c763d5-1fbd-428a-bf62-cb4102fbc145/volumes/kubernetes.io~projected/kube-api-access-g5gp8 DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95020920-a029-4c17-8ce1-e66a4982a00d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes/kubernetes.io~projected/kube-api-access-rrjh6 DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84d699e8-7ccc-4e2c-b6c6-e309946bee2e/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/362105ce-9f40-4ad3-964b-472512bf9518/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1bd4c6b277ce6a8 MacAddress:6e:32:61:31:9e:2a Speed:10000 Mtu:8901} {Name:390e6e4d5813fba MacAddress:0a:2e:04:f1:7b:97 Speed:10000 Mtu:8901} {Name:4a55e5db4901f63 MacAddress:12:90:53:a7:11:7f Speed:10000 Mtu:8901} {Name:52b04e9ef70e2f6 MacAddress:2a:b0:d6:17:c7:4c Speed:10000 Mtu:8901} {Name:6a85974966c3320 MacAddress:22:88:77:46:66:5d Speed:10000 Mtu:8901} {Name:8348e72ade41ce5 MacAddress:26:de:dc:32:5d:5f Speed:10000 Mtu:8901} {Name:928086259a3ecfd MacAddress:1a:46:2d:3d:ba:d4 Speed:10000 Mtu:8901} {Name:987d14d3adfd8f0 MacAddress:ee:a1:e3:b7:90:b9 Speed:10000 Mtu:8901} {Name:a9f66f322326a02 MacAddress:aa:b9:dd:5b:eb:0b Speed:10000 Mtu:8901} {Name:bbb42e5d5654672 MacAddress:ca:8c:e7:77:8c:2a Speed:10000 Mtu:8901} {Name:be120e3bc42ee7c MacAddress:7a:92:c0:48:a9:86 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:01:b1:50:73:7f Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:07:c7:2c:5a:62 Speed:0 Mtu:8901} {Name:d0bab689ae469a9 MacAddress:2a:b8:9e:7c:39:e4 Speed:10000 Mtu:8901} {Name:d5b091b0bb1fd31 MacAddress:ee:70:f8:f2:0e:e3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:01:b1:50:73:7f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:89:e1:a6:f6:ad Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ba:b1:27:d6:8e:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.125509 4925 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.125553 4925 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.125835 4925 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.125856 4925 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-184.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.125990 4925 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.125997 4925 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126022 4925 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126046 4925 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:59:00.126182 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126136 4925 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 06:59:00.126580 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126362 4925 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 06:59:00.126580 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126466 4925 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 06:59:00.126580 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126502 4925 kubelet.go:493] "Attempting to sync node with API server" Nov 03 06:59:00.126580 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126512 4925 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 06:59:00.126580 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126522 4925 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 06:59:00.126580 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126531 4925 kubelet.go:405] "Adding apiserver pod source" Nov 03 06:59:00.126580 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.126538 4925 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 06:59:00.127233 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.127220 4925 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 06:59:00.127233 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.127234 4925 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:59:00.128933 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.128114 4925 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 06:59:00.139976 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.139960 4925 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 06:59:00.139976 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.139975 4925 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:59:00.140158 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140146 4925 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 06:59:00.140361 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140349 4925 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 06:59:00.140392 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140380 4925 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 06:59:00.140494 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140485 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 06:59:00.140522 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140501 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 06:59:00.140522 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140508 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 06:59:00.140522 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140515 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140525 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140534 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140540 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140546 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140553 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140559 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140567 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140577 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 06:59:00.140604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140603 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 06:59:00.140841 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.140613 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 06:59:00.144150 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.144137 4925 server.go:1263] "Started kubelet" Nov 03 06:59:00.144245 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.144213 4925 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 06:59:00.144645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.144579 4925 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 06:59:00.144747 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.144654 4925 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 06:59:00.144704 ip-10-0-136-184 systemd[1]: Started Kubernetes Kubelet. Nov 03 06:59:00.145353 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.145319 4925 server.go:317] "Adding debug handlers to kubelet server" Nov 03 06:59:00.145522 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.145462 4925 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 06:59:00.147960 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.147943 4925 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 06:59:00.148695 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.148315 4925 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:59:00.148695 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.148367 4925 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 06:59:00.148695 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.148379 4925 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 06:59:00.148695 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.148536 4925 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 06:59:00.151831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.149069 4925 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 06:59:00.151831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.149301 4925 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:21 +0000 UTC" deadline="2028-07-03 03:09:58.161111919 +0000 UTC" Nov 03 06:59:00.151831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.149336 4925 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14588h10m58.011778633s" Nov 03 06:59:00.151831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.151353 4925 factory.go:55] Registering systemd factory Nov 03 06:59:00.151831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.151372 4925 factory.go:223] Registration of the systemd container factory successfully Nov 03 06:59:00.152154 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.151866 4925 factory.go:153] Registering CRI-O factory Nov 03 06:59:00.152154 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.151885 4925 factory.go:223] Registration of the crio container factory successfully Nov 03 06:59:00.152154 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.151968 4925 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 06:59:00.152154 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.151989 4925 factory.go:103] Registering Raw factory Nov 03 06:59:00.152666 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.152642 4925 manager.go:1199] Started watching for new ooms in manager Nov 03 06:59:00.153047 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.152948 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25276ee9-63e0-4d32-834e-8223c741b724" volumeName="kubernetes.io/projected/25276ee9-63e0-4d32-834e-8223c741b724-kube-api-access-b9qpj" seLinuxMountContext="" Nov 03 06:59:00.153047 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.152995 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="452ab275-c3b0-4c31-8603-7af7cd666346" volumeName="kubernetes.io/projected/452ab275-c3b0-4c31-8603-7af7cd666346-kube-api-access-2k2xn" seLinuxMountContext="" Nov 03 06:59:00.154294 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154263 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580a0715-a63d-43ef-983c-6d459af6ba42" volumeName="kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-multus-daemon-config" seLinuxMountContext="" Nov 03 06:59:00.154414 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154296 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f55630a1-1afe-446c-bda0-20bfba394fb5" volumeName="kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates" seLinuxMountContext="" Nov 03 06:59:00.154414 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154310 4925 manager.go:319] Starting recovery of all containers Nov 03 06:59:00.154414 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154311 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063967c5-741c-46f1-a023-9384b59ccaa7" volumeName="kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets" seLinuxMountContext="" Nov 03 06:59:00.154414 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154391 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dbc159e-0969-417d-91fb-819d526579a3" volumeName="kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 06:59:00.154414 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154405 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902e5dbc-172c-496e-afaf-2e847ba1f852" volumeName="kubernetes.io/configmap/902e5dbc-172c-496e-afaf-2e847ba1f852-telemetry-config" seLinuxMountContext="" Nov 03 06:59:00.154414 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154417 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9627c15-ddb9-4ca1-b1d1-606c8c0453b3" volumeName="kubernetes.io/projected/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-kube-api-access-cj2xf" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154433 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f55630a1-1afe-446c-bda0-20bfba394fb5" volumeName="kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154445 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dbc159e-0969-417d-91fb-819d526579a3" volumeName="kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154456 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="362105ce-9f40-4ad3-964b-472512bf9518" volumeName="kubernetes.io/projected/362105ce-9f40-4ad3-964b-472512bf9518-kube-api-access-n2kh4" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154469 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728637b6-bbb0-4532-adde-4555c4f4153d" volumeName="kubernetes.io/projected/728637b6-bbb0-4532-adde-4555c4f4153d-kube-api-access-qscpq" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154481 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728637b6-bbb0-4532-adde-4555c4f4153d" volumeName="kubernetes.io/secret/728637b6-bbb0-4532-adde-4555c4f4153d-serving-cert" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154494 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8346ef93-89cc-47f7-b3bf-3a309b6efa2d" volumeName="kubernetes.io/projected/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-kube-api-access-ch4qf" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154516 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8346ef93-89cc-47f7-b3bf-3a309b6efa2d" volumeName="kubernetes.io/secret/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-cert" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154529 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="452ab275-c3b0-4c31-8603-7af7cd666346" volumeName="kubernetes.io/secret/452ab275-c3b0-4c31-8603-7af7cd666346-serving-cert" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154540 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d699e8-7ccc-4e2c-b6c6-e309946bee2e" volumeName="kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-stats-auth" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154554 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e15cc64f-4c50-4c6d-89fd-d2144ad0d38c" volumeName="kubernetes.io/empty-dir/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-tmp-dir" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154566 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6eb01e1-380a-468c-8097-d34540808454" volumeName="kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-config" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154578 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f55630a1-1afe-446c-bda0-20bfba394fb5" volumeName="kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154591 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bee72bc7-52f6-45a6-8118-eefba66e6992" volumeName="kubernetes.io/configmap/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-config" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154602 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6b39d84-7483-46fb-ab4b-d8de4e3a1212" volumeName="kubernetes.io/configmap/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-iptables-alerter-script" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154614 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6eb01e1-380a-468c-8097-d34540808454" volumeName="kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-script-lib" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154646 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7781680-4a88-4a3f-8fc1-61c702b89f63" volumeName="kubernetes.io/configmap/f7781680-4a88-4a3f-8fc1-61c702b89f63-konnectivity-ca" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154658 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="362105ce-9f40-4ad3-964b-472512bf9518" volumeName="kubernetes.io/secret/362105ce-9f40-4ad3-964b-472512bf9518-serving-cert" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154671 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e7d1d1e-2fb6-412d-8597-f6f4882d0389" volumeName="kubernetes.io/projected/4e7d1d1e-2fb6-412d-8597-f6f4882d0389-kube-api-access-cpp4t" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154686 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c763d5-1fbd-428a-bf62-cb4102fbc145" volumeName="kubernetes.io/projected/66c763d5-1fbd-428a-bf62-cb4102fbc145-kube-api-access-g5gp8" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154697 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95020920-a029-4c17-8ce1-e66a4982a00d" volumeName="kubernetes.io/projected/95020920-a029-4c17-8ce1-e66a4982a00d-kube-api-access-p4f7r" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154713 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e15cc64f-4c50-4c6d-89fd-d2144ad0d38c" volumeName="kubernetes.io/projected/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-kube-api-access-pdzkl" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154725 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab2a8b5-c6f2-45df-9fcc-161d6ad92498" volumeName="kubernetes.io/projected/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-kube-api-access-2glrf" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154736 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f55630a1-1afe-446c-bda0-20bfba394fb5" volumeName="kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154751 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bfb1d47-8ee3-4e09-a4ea-ddcef2244233" volumeName="kubernetes.io/projected/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233-kube-api-access-gnhbg" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154765 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fc2a19a-80f8-4318-a6fd-b246b3022d0e" volumeName="kubernetes.io/configmap/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-serviceca" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154776 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca92ce4d-7cf4-4da5-8609-920db260c2d3" volumeName="kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154795 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f55630a1-1afe-446c-bda0-20bfba394fb5" volumeName="kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154807 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7781680-4a88-4a3f-8fc1-61c702b89f63" volumeName="kubernetes.io/secret/f7781680-4a88-4a3f-8fc1-61c702b89f63-agent-certs" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154819 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271984b8-d201-41a3-a545-690f8a9fad89" volumeName="kubernetes.io/projected/271984b8-d201-41a3-a545-690f8a9fad89-kube-api-access-q7c5q" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154834 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="452ab275-c3b0-4c31-8603-7af7cd666346" volumeName="kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-config" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154846 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e15cc64f-4c50-4c6d-89fd-d2144ad0d38c" volumeName="kubernetes.io/configmap/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-config-volume" seLinuxMountContext="" Nov 03 06:59:00.154847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154861 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f55630a1-1afe-446c-bda0-20bfba394fb5" volumeName="kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154874 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063967c5-741c-46f1-a023-9384b59ccaa7" volumeName="kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154885 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dbc159e-0969-417d-91fb-819d526579a3" volumeName="kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-binary-copy" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154897 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271984b8-d201-41a3-a545-690f8a9fad89" volumeName="kubernetes.io/secret/271984b8-d201-41a3-a545-690f8a9fad89-serving-cert" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154909 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728637b6-bbb0-4532-adde-4555c4f4153d" volumeName="kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-service-ca-bundle" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154923 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728637b6-bbb0-4532-adde-4555c4f4153d" volumeName="kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-trusted-ca-bundle" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154935 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6eb01e1-380a-468c-8097-d34540808454" volumeName="kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-env-overrides" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154947 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6eb01e1-380a-468c-8097-d34540808454" volumeName="kubernetes.io/secret/e6eb01e1-380a-468c-8097-d34540808454-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154959 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab2a8b5-c6f2-45df-9fcc-161d6ad92498" volumeName="kubernetes.io/empty-dir/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-tmp-dir" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154970 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063967c5-741c-46f1-a023-9384b59ccaa7" volumeName="kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154983 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6b39d84-7483-46fb-ab4b-d8de4e3a1212" volumeName="kubernetes.io/projected/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-kube-api-access-bh76m" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.154995 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580a0715-a63d-43ef-983c-6d459af6ba42" volumeName="kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-cni-binary-copy" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155025 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95020920-a029-4c17-8ce1-e66a4982a00d" volumeName="kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-etc-tuned" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155039 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="018a989a-6542-4db1-bd2a-d80df7bbed24" volumeName="kubernetes.io/empty-dir/018a989a-6542-4db1-bd2a-d80df7bbed24-data-volume" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155051 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271984b8-d201-41a3-a545-690f8a9fad89" volumeName="kubernetes.io/configmap/271984b8-d201-41a3-a545-690f8a9fad89-config" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155062 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728637b6-bbb0-4532-adde-4555c4f4153d" volumeName="kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-snapshots" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155073 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="936c5f59-3e71-4d99-ad3a-b0762bf8afda" volumeName="kubernetes.io/projected/936c5f59-3e71-4d99-ad3a-b0762bf8afda-kube-api-access-z2pw6" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155096 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6eb01e1-380a-468c-8097-d34540808454" volumeName="kubernetes.io/projected/e6eb01e1-380a-468c-8097-d34540808454-kube-api-access-tc9m9" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155108 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="018a989a-6542-4db1-bd2a-d80df7bbed24" volumeName="kubernetes.io/projected/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-api-access-tnfg9" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155119 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063967c5-741c-46f1-a023-9384b59ccaa7" volumeName="kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155131 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10e402a1-d12e-4559-aae4-38ca72e7f1cf" volumeName="kubernetes.io/projected/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kube-api-access-ckqdw" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155142 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="362105ce-9f40-4ad3-964b-472512bf9518" volumeName="kubernetes.io/configmap/362105ce-9f40-4ad3-964b-472512bf9518-config" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155155 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d699e8-7ccc-4e2c-b6c6-e309946bee2e" volumeName="kubernetes.io/projected/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-kube-api-access-cv2vm" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155168 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d699e8-7ccc-4e2c-b6c6-e309946bee2e" volumeName="kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-default-certificate" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155179 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902e5dbc-172c-496e-afaf-2e847ba1f852" volumeName="kubernetes.io/projected/902e5dbc-172c-496e-afaf-2e847ba1f852-kube-api-access-pc487" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155192 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95020920-a029-4c17-8ce1-e66a4982a00d" volumeName="kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-tmp" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155204 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063967c5-741c-46f1-a023-9384b59ccaa7" volumeName="kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155215 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f82f59-91bd-4845-ae6b-ca143189342d" volumeName="kubernetes.io/configmap/88f82f59-91bd-4845-ae6b-ca143189342d-nginx-conf" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155226 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bee72bc7-52f6-45a6-8118-eefba66e6992" volumeName="kubernetes.io/projected/bee72bc7-52f6-45a6-8118-eefba66e6992-kube-api-access-rdxnh" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155241 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e47cc873-c3fe-421e-8882-9ce4b4ff3737" volumeName="kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155253 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063967c5-741c-46f1-a023-9384b59ccaa7" volumeName="kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155265 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c1e3ef-149c-401f-8e88-8bdafa0faae6" volumeName="kubernetes.io/projected/40c1e3ef-149c-401f-8e88-8bdafa0faae6-kube-api-access-blqd2" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155276 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="452ab275-c3b0-4c31-8603-7af7cd666346" volumeName="kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-trusted-ca" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155287 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f55630a1-1afe-446c-bda0-20bfba394fb5" volumeName="kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155298 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063967c5-741c-46f1-a023-9384b59ccaa7" volumeName="kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155311 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dbc159e-0969-417d-91fb-819d526579a3" volumeName="kubernetes.io/projected/0dbc159e-0969-417d-91fb-819d526579a3-kube-api-access-qtjv9" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155326 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fc2a19a-80f8-4318-a6fd-b246b3022d0e" volumeName="kubernetes.io/projected/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-kube-api-access-sffvz" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155337 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580a0715-a63d-43ef-983c-6d459af6ba42" volumeName="kubernetes.io/projected/580a0715-a63d-43ef-983c-6d459af6ba42-kube-api-access-wxfgw" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155353 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728637b6-bbb0-4532-adde-4555c4f4153d" volumeName="kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-tmp" seLinuxMountContext="" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155365 4925 reconstruct.go:97] "Volume reconstruction finished" Nov 03 06:59:00.156273 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.155372 4925 reconciler.go:29] "Reconciler: start to sync state" Nov 03 06:59:00.157982 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.157822 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/1.log" Nov 03 06:59:00.159109 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.159088 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/1.log" Nov 03 06:59:00.161513 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.161494 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 06:59:00.162516 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.162499 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 06:59:00.181326 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.181287 4925 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 06:59:00.182715 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.182692 4925 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 06:59:00.182715 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.182717 4925 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 06:59:00.182860 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.182738 4925 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 06:59:00.182860 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.182778 4925 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 06:59:00.209546 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.209480 4925 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="14bd3d450ede31cfcfc3d49dd585d61f13f7b274e1aa20eba5985cfb2327becb" exitCode=0 Nov 03 06:59:00.209546 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.209506 4925 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="50645334841fad19443b7bb21695330912f95737d20a710a7a0e1e101c0e4b32" exitCode=0 Nov 03 06:59:00.209546 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.209513 4925 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="149349693e4d9dc9400ce7142dbe637166ca98bac122dd86be259f468c2ea3e1" exitCode=0 Nov 03 06:59:00.209546 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.209518 4925 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="2491991c6c6b043dde46c42d8a72829bb31b0559c5f420c663b27cc33405076c" exitCode=0 Nov 03 06:59:00.209546 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.209523 4925 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="4be6b0c47e79f8a4dd66ceb6cd71228392a25291456cecd464ba0e237334201f" exitCode=0 Nov 03 06:59:00.209546 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.209529 4925 generic.go:356] "Generic (PLEG): container finished" podID="0dbc159e-0969-417d-91fb-819d526579a3" containerID="4d030c626ab7f8d5beae44262ef0c601ca2f37f2462f08cfc45f8c31f2d5892b" exitCode=0 Nov 03 06:59:00.216094 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.216066 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 06:59:00.216420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.216395 4925 generic.go:356] "Generic (PLEG): container finished" podID="e6eb01e1-380a-468c-8097-d34540808454" containerID="c3e80a2b6b1c29718af15a534036a799f0a20cdcd185bbe9a38e7ddd1473f129" exitCode=1 Nov 03 06:59:00.224725 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.224693 4925 generic.go:356] "Generic (PLEG): container finished" podID="c15c975752830b5906436794da287b80" containerID="ded14717958d4b1b1b4687ffd6b41dffaa142318c3bf4c973350d91aae8be084" exitCode=0 Nov 03 06:59:00.231806 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.231788 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/1.log" Nov 03 06:59:00.231899 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.231816 4925 generic.go:356] "Generic (PLEG): container finished" podID="452ab275-c3b0-4c31-8603-7af7cd666346" containerID="ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3" exitCode=255 Nov 03 06:59:00.239643 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.239626 4925 manager.go:324] Recovery completed Nov 03 06:59:00.248841 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.248825 4925 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:59:00.258197 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.258177 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:00.259456 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.259440 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:00.259524 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.259468 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:00.259524 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.259478 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:00.262114 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262100 4925 cpu_manager.go:226] "Starting" policy="none" Nov 03 06:59:00.262114 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262112 4925 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 06:59:00.262220 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262129 4925 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 06:59:00.262256 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262112 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:00.262256 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262237 4925 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 06:59:00.262315 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262248 4925 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 06:59:00.262315 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262265 4925 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 06:59:00.262315 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262270 4925 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 06:59:00.262315 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262276 4925 policy_none.go:50] "Start" Nov 03 06:59:00.262315 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262283 4925 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 06:59:00.262315 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262292 4925 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 06:59:00.262493 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262434 4925 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 06:59:00.262493 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262445 4925 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 06:59:00.262493 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.262453 4925 policy_none.go:44] "Start" Nov 03 06:59:00.263720 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.263703 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:00.263776 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.263735 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:00.263776 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.263749 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:00.267063 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.267035 4925 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 06:59:00.269569 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.269557 4925 manager.go:341] "Starting Device Plugin manager" Nov 03 06:59:00.269638 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.269580 4925 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 06:59:00.269638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.269589 4925 server.go:87] "Starting device plugin registration server" Nov 03 06:59:00.269880 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.269868 4925 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 06:59:00.269910 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.269884 4925 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 06:59:00.269996 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.269956 4925 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 06:59:00.270151 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.270106 4925 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 06:59:00.270235 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.270197 4925 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 06:59:00.270235 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.270206 4925 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 06:59:00.270399 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.270303 4925 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T06:59:00.270182636Z","UUID":"6aa12d7e-6033-4dfe-8d5d-7f6d3b24441b","Handler":null,"Name":"","Endpoint":""} Nov 03 06:59:00.271481 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.271462 4925 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 06:59:00.271587 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.271500 4925 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" Nov 03 06:59:00.271720 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.271704 4925 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 06:59:00.271823 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.271758 4925 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 06:59:00.283421 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.283382 4925 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal"] Nov 03 06:59:00.283526 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.283515 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:00.284498 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.284473 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:00.284498 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.284501 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:00.284604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.284512 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:00.284806 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.284796 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:00.284928 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.284917 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:00.285557 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285530 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:00.285638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285562 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:00.285638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285563 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:00.285638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285577 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:00.285638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285588 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:00.285638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285600 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:00.285884 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.285730 4925 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.285983 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285954 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"1f565955bc42ade3fa3b9ef6d6540a53","Type":"ContainerStarted","Data":"ed1d537994cc17d0f766a467fc4b83670e786c952917a776991f34c5baddc1ab"} Nov 03 06:59:00.286077 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.285994 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"1f565955bc42ade3fa3b9ef6d6540a53","Type":"ContainerStarted","Data":"6c3ef26a26205be9c8fcedccc9fded07e405e2f6cfe43260be73b2fcab76c1ce"} Nov 03 06:59:00.286077 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.286059 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:00.286171 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.286111 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"c15c975752830b5906436794da287b80","Type":"ContainerStarted","Data":"11f92ef661280d5617390c852bb68938d3c1a260ea742d21ae3d4ea805882aee"} Nov 03 06:59:00.286171 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.286133 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"c15c975752830b5906436794da287b80","Type":"ContainerDied","Data":"ded14717958d4b1b1b4687ffd6b41dffaa142318c3bf4c973350d91aae8be084"} Nov 03 06:59:00.286171 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.286152 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" event={"ID":"c15c975752830b5906436794da287b80","Type":"ContainerStarted","Data":"cfa685298648eee3f11f06e35de973b57e82fa391e3a1042fc91013154402c32"} Nov 03 06:59:00.286780 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.286766 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:00.286855 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.286794 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:00.286855 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.286809 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:00.286969 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.286956 4925 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-184.us-west-1.compute.internal\" not found" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.342544 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.342522 4925 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 06:59:00.345212 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.345195 4925 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:00.345279 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.345204 4925 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:00.345279 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.345249 4925 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:00.349450 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.349435 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.370604 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.370584 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:00.371523 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.371502 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:00.371586 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.371533 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:00.371586 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.371544 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:00.371662 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.371588 4925 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.396219 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.396164 4925 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:00.436797 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.436778 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:00.436885 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.436819 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.436885 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.436833 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.443063 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.443044 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.443147 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.443070 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.443189 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.443167 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f565955bc42ade3fa3b9ef6d6540a53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"1f565955bc42ade3fa3b9ef6d6540a53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.490680 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.490631 4925 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.490748 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.490689 4925 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.491236 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.491219 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeNotReady" Nov 03 06:59:00.491341 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.491277 4925 setters.go:546] "Node became not ready" node="ip-10-0-136-184.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T06:59:00Z","lastTransitionTime":"2026-11-03T06:59:00Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 06:59:00.515449 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.515428 4925 nodeinfomanager.go:471] existing CSINode "ip-10-0-136-184.us-west-1.compute.internal" is owned by different node (oldNodeID="4c764cc0-2d92-4256-93df-483fbf40a6c0", newNodeID=""), cleaning up... Nov 03 06:59:00.517427 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.517411 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:00.517493 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:00.517453 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.543618 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.543594 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.543618 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.543607 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.543719 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.543634 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.543719 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.543658 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15c975752830b5906436794da287b80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"c15c975752830b5906436794da287b80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.543719 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.543657 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f565955bc42ade3fa3b9ef6d6540a53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"1f565955bc42ade3fa3b9ef6d6540a53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.543719 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.543693 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f565955bc42ade3fa3b9ef6d6540a53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" (UID: \"1f565955bc42ade3fa3b9ef6d6540a53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:00.763202 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.763137 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-184.us-west-1.compute.internal" event="NodeReady" Nov 03 06:59:00.763318 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:00.763218 4925 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 06:59:01.127565 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.127538 4925 apiserver.go:52] "Watching apiserver" Nov 03 06:59:01.239603 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.239577 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:01.239738 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.239678 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:01.306238 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.306203 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:01.306238 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.306218 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:01.306919 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.306253 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:01.306919 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.306256 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal" Nov 03 06:59:01.306919 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.306594 4925 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 06:59:01.307275 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.307252 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57998bb766-8gfrg","openshift-network-operator/iptables-alerter-vn65v","openshift-ingress/router-default-699757bbb7-drfpj","openshift-ovn-kubernetes/ovnkube-node-pb8jn","openshift-insights/insights-operator-7c4986d449-299kl","openshift-console-operator/console-operator-7cd56b8dbc-ldktd","openshift-multus/network-metrics-daemon-g8m65","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.us-west-1.compute.internal","openshift-image-registry/image-registry-54fc5dcf5f-l5q5j","openshift-network-console/networking-console-plugin-86c478bb88-lpk6z","kube-system/kas-connection-checker-685895c7c9-hzqt4","openshift-cluster-node-tuning-operator/tuned-zhjnj","openshift-image-registry/node-ca-9vk5q","openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v","openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6","kube-system/global-pull-secret-syncer-dtzxb","kube-system/kas-connection-checker-685895c7c9-4jk57","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4","openshift-multus/multus-additional-cni-plugins-x5cxd","openshift-dns/dns-default-cpnw7","openshift-ingress-canary/ingress-canary-fstk4","openshift-network-diagnostics/network-check-target-dfpc7","openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5","kube-system/konnectivity-agent-4gwnd","openshift-dns/node-resolver-zq5qq","kube-system/kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb","openshift-insights/insights-runtime-extractor-jjq57","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz","openshift-multus/multus-v9wll","openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd","kube-system/kas-connection-checker-685895c7c9-5ndhb"] Nov 03 06:59:01.308295 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.308280 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.312258 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.312229 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:01.313408 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.313387 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:01.313974 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.313953 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.314858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.314844 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:01.317992 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.317976 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:01.319038 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.319004 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.321705 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.321690 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.322328 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.322311 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.350663 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.350642 4925 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 06:59:01.363510 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.363486 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.363614 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.363598 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.363669 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.363598 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 06:59:01.363945 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.363928 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.364058 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.363962 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 06:59:01.364058 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.363999 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.364058 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.364003 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.364219 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.364004 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.364219 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.364208 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.364313 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.364286 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 06:59:01.364610 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.364597 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.365163 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.365149 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 06:59:01.365423 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.365401 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 06:59:01.365570 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.365554 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 06:59:01.365683 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.365668 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 06:59:01.365746 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.365705 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8gp29\"" Nov 03 06:59:01.365876 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.365861 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 06:59:01.417399 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.417324 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 06:59:01.417399 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.417361 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 06:59:01.418292 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.418268 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 06:59:01.449390 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449366 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffvz\" (UniqueName: \"kubernetes.io/projected/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-kube-api-access-sffvz\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:59:01.449487 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449398 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-cnibin\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.449487 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449420 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:01.449487 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449447 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.449590 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449516 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.449651 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449634 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/018a989a-6542-4db1-bd2a-d80df7bbed24-crio-socket\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.449683 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449666 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.449711 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449695 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.449834 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449817 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-hostroot\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.449876 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449858 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-script-lib\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.449909 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449873 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.449909 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449896 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/e6eb01e1-380a-468c-8097-d34540808454-kube-api-access-tc9m9\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.449962 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449924 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-service-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.449962 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449943 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.450053 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449961 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:01.450053 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449984 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-dbus\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.450053 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.449999 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-registration-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.450053 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450042 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-system-cni-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.450211 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450085 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.450211 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450126 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-service-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.450211 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450143 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.450211 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450173 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch4qf\" (UniqueName: \"kubernetes.io/projected/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-kube-api-access-ch4qf\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:59:01.450211 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450178 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.450352 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450321 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:01.450352 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450343 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-socket-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.450405 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450359 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.450405 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450373 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.450458 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450402 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-system-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.450458 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450441 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.450513 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450457 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-netns\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.450513 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450484 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362105ce-9f40-4ad3-964b-472512bf9518-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:59:01.450513 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450499 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-tmp\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.450593 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450541 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-var-lib-kubelet\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.450593 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450575 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-device-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.450658 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450598 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-systemd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.450658 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450610 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-tmp\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.450658 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450614 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-etc-tuned\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.450739 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450663 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2kh4\" (UniqueName: \"kubernetes.io/projected/362105ce-9f40-4ad3-964b-472512bf9518-kube-api-access-n2kh4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:59:01.450739 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450668 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-etc-tuned\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.450739 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450707 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.450819 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450748 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-cert\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:59:01.450881 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450868 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/902e5dbc-172c-496e-afaf-2e847ba1f852-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:01.450914 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450898 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/018a989a-6542-4db1-bd2a-d80df7bbed24-data-volume\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.450985 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450970 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/018a989a-6542-4db1-bd2a-d80df7bbed24-data-volume\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.451033 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.450990 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzkl\" (UniqueName: \"kubernetes.io/projected/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-kube-api-access-pdzkl\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.451071 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451044 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-systemd\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.451071 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451065 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqd2\" (UniqueName: \"kubernetes.io/projected/40c1e3ef-149c-401f-8e88-8bdafa0faae6-kube-api-access-blqd2\") pod \"kas-connection-checker-685895c7c9-hzqt4\" (UID: \"40c1e3ef-149c-401f-8e88-8bdafa0faae6\") " pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" Nov 03 06:59:01.451137 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451082 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/728637b6-bbb0-4532-adde-4555c4f4153d-serving-cert\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.451137 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451105 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7781680-4a88-4a3f-8fc1-61c702b89f63-agent-certs\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:59:01.451137 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451128 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.451277 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451150 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6eb01e1-380a-468c-8097-d34540808454-ovn-node-metrics-cert\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.451277 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451179 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh76m\" (UniqueName: \"kubernetes.io/projected/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-kube-api-access-bh76m\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:59:01.451277 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451204 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-systemd-units\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.451277 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451235 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-etc-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.451277 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451261 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-config\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.451502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451274 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/728637b6-bbb0-4532-adde-4555c4f4153d-serving-cert\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.451502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451299 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/66c763d5-1fbd-428a-bf62-cb4102fbc145-kube-api-access-g5gp8\") pod \"volume-data-source-validator-58c9f5f46d-z4nr5\" (UID: \"66c763d5-1fbd-428a-bf62-cb4102fbc145\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" Nov 03 06:59:01.451502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451329 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362105ce-9f40-4ad3-964b-472512bf9518-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:59:01.451502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451447 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/728637b6-bbb0-4532-adde-4555c4f4153d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.451502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451446 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-cnibin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.451502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451451 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-config\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.451502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451486 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-netns\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451540 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-etc-kubernetes\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451576 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451619 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-multus-daemon-config\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451647 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451672 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-kubelet-config\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451703 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhbg\" (UniqueName: \"kubernetes.io/projected/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233-kube-api-access-gnhbg\") pod \"migrator-66df5d496b-29ktd\" (UID: \"1bfb1d47-8ee3-4e09-a4ea-ddcef2244233\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451730 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjv9\" (UniqueName: \"kubernetes.io/projected/0dbc159e-0969-417d-91fb-819d526579a3-kube-api-access-qtjv9\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451756 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-tmp-dir\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451783 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88f82f59-91bd-4845-ae6b-ca143189342d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:01.451820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451809 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452ab275-c3b0-4c31-8603-7af7cd666346-serving-cert\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451834 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7c5q\" (UniqueName: \"kubernetes.io/projected/271984b8-d201-41a3-a545-690f8a9fad89-kube-api-access-q7c5q\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451861 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7781680-4a88-4a3f-8fc1-61c702b89f63-konnectivity-ca\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451887 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2xn\" (UniqueName: \"kubernetes.io/projected/452ab275-c3b0-4c31-8603-7af7cd666346-kube-api-access-2k2xn\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451889 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-tmp-dir\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451914 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-lib-modules\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451939 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f7r\" (UniqueName: \"kubernetes.io/projected/95020920-a029-4c17-8ce1-e66a4982a00d-kube-api-access-p4f7r\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451977 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452000 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452ab275-c3b0-4c31-8603-7af7cd666346-serving-cert\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.451999 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrjh6\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452067 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-serviceca\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452085 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-sys-fs\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452111 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-bin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452187 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-conf\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452232 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-run\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.452270 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452279 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-host-slash\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452296 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-multus\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452322 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfgw\" (UniqueName: \"kubernetes.io/projected/580a0715-a63d-43ef-983c-6d459af6ba42-kube-api-access-wxfgw\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452344 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-netd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452363 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-config\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452380 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452321 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-serviceca\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452408 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452425 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-socket-dir-parent\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452444 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-env-overrides\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452469 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-stats-auth\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452505 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452524 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452541 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qscpq\" (UniqueName: \"kubernetes.io/projected/728637b6-bbb0-4532-adde-4555c4f4153d-kube-api-access-qscpq\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452564 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-cni-binary-copy\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452590 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-multus-certs\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452612 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-slash\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452652 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-config-volume\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452695 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.452741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452725 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-default-certificate\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452762 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452785 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-node-log\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452807 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2vm\" (UniqueName: \"kubernetes.io/projected/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-kube-api-access-cv2vm\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452837 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452874 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-tmp\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452905 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-host\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452945 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95020920-a029-4c17-8ce1-e66a4982a00d-tmp\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452966 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxnh\" (UniqueName: \"kubernetes.io/projected/bee72bc7-52f6-45a6-8118-eefba66e6992-kube-api-access-rdxnh\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.452990 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-log-socket\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453023 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-modprobe-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453042 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qpj\" (UniqueName: \"kubernetes.io/projected/25276ee9-63e0-4d32-834e-8223c741b724-kube-api-access-b9qpj\") pod \"kas-connection-checker-685895c7c9-5ndhb\" (UID: \"25276ee9-63e0-4d32-834e-8223c741b724\") " pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453141 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453232 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.453252 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453249 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-kubelet\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453284 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453311 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453323 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-bin\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453341 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysconfig\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453368 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pw6\" (UniqueName: \"kubernetes.io/projected/936c5f59-3e71-4d99-ad3a-b0762bf8afda-kube-api-access-z2pw6\") pod \"network-check-source-6f78d59f99-fb26v\" (UID: \"936c5f59-3e71-4d99-ad3a-b0762bf8afda\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453390 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453430 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/271984b8-d201-41a3-a545-690f8a9fad89-serving-cert\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:59:01.453642 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453563 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-conf-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453641 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-ovn\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453674 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453699 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-hosts-file\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453788 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/271984b8-d201-41a3-a545-690f8a9fad89-config\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453808 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-iptables-alerter-script\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453817 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453823 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.453858 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453845 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9wc\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453870 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glrf\" (UniqueName: \"kubernetes.io/projected/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-kube-api-access-2glrf\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453898 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqdw\" (UniqueName: \"kubernetes.io/projected/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kube-api-access-ckqdw\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453915 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-var-lib-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453933 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpp4t\" (UniqueName: \"kubernetes.io/projected/4e7d1d1e-2fb6-412d-8597-f6f4882d0389-kube-api-access-cpp4t\") pod \"kas-connection-checker-685895c7c9-4jk57\" (UID: \"4e7d1d1e-2fb6-412d-8597-f6f4882d0389\") " pod="kube-system/kas-connection-checker-685895c7c9-4jk57" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453956 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2xf\" (UniqueName: \"kubernetes.io/projected/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-kube-api-access-cj2xf\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.453984 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-snapshots\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454027 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-kubelet\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454051 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-kubernetes\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454080 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc487\" (UniqueName: \"kubernetes.io/projected/902e5dbc-172c-496e-afaf-2e847ba1f852-kube-api-access-pc487\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454130 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfg9\" (UniqueName: \"kubernetes.io/projected/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-api-access-tnfg9\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454145 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454180 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454215 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-sys\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454217 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/728637b6-bbb0-4532-adde-4555c4f4153d-snapshots\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.454248 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454217 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454260 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454304 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454339 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454341 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454428 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-trusted-ca\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454518 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454551 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454581 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-k8s-cni-cncf-io\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454607 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454609 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-tmp-dir\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454647 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454657 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-host\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454673 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-tmp-dir\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454756 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/452ab275-c3b0-4c31-8603-7af7cd666346-trusted-ca\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.454816 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.454780 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.464161 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464145 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 06:59:01.464161 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464155 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 06:59:01.464260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464188 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.464498 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464487 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 06:59:01.464563 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464545 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 06:59:01.464645 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464585 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.464707 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464608 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.464707 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464615 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.464707 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464649 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 06:59:01.464845 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464718 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 06:59:01.464845 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464744 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.464845 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464771 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 06:59:01.464845 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464793 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.464845 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464796 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.464845 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464814 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 06:59:01.464845 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.464845 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.465100 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465075 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.465100 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465080 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465108 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465098 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465143 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465146 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465160 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465181 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465217 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465222 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465221 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.465260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465219 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 06:59:01.465747 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465519 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 06:59:01.465747 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465588 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.465840 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465782 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 06:59:01.465840 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465797 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465840 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465834 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.465840 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465843 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.465840 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.465846 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 06:59:01.466162 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466150 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 06:59:01.466296 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466215 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.466296 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466247 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 06:59:01.466296 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466281 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 06:59:01.466477 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466332 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.466477 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466396 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 06:59:01.466477 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466416 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.466477 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466416 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.466477 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466430 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.466477 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466435 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 06:59:01.466736 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466658 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.466736 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466661 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.466736 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466709 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 06:59:01.466972 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466742 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 06:59:01.466972 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466757 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:01.466972 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466794 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 06:59:01.466972 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466897 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 06:59:01.466972 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466910 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 06:59:01.466972 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.466913 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 06:59:01.467397 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467059 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 06:59:01.467397 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467070 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 06:59:01.467397 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467078 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 06:59:01.467397 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467109 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6z6wp\"" Nov 03 06:59:01.467397 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467202 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 06:59:01.467397 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467329 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9lht\"" Nov 03 06:59:01.467397 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467395 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 06:59:01.467705 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467427 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5znfn\"" Nov 03 06:59:01.467705 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467437 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tq2gc\"" Nov 03 06:59:01.467705 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467464 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftjg\"" Nov 03 06:59:01.467705 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467511 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lnjtr\"" Nov 03 06:59:01.467705 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467673 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467714 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467721 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dznd\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467726 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xbxhv\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467730 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p8ttv\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467724 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467744 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kl8t2\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467772 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxfbc\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467780 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nd4b5\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467787 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-htlhr\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467785 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dmgt\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467808 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kx4sk\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467821 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zhxgw\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467822 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lr4sn\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467834 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8z7js\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467842 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9n2tm\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467847 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44mbd\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467836 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467857 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xp6sr\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467869 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8c889\"" Nov 03 06:59:01.467955 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.467903 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-br79x\"" Nov 03 06:59:01.470207 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.470190 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:01.470285 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.470271 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-script-lib\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.471375 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471348 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362105ce-9f40-4ad3-964b-472512bf9518-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:59:01.471465 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471403 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-cert\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:59:01.471465 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471426 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.471465 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471427 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/902e5dbc-172c-496e-afaf-2e847ba1f852-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:01.471599 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471467 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.471599 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471474 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6eb01e1-380a-468c-8097-d34540808454-ovn-node-metrics-cert\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.471599 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471500 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7781680-4a88-4a3f-8fc1-61c702b89f63-agent-certs\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:59:01.471599 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.471519 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362105ce-9f40-4ad3-964b-472512bf9518-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472726 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-ovnkube-config\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472760 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7781680-4a88-4a3f-8fc1-61c702b89f63-konnectivity-ca\") pod \"konnectivity-agent-4gwnd\" (UID: \"f7781680-4a88-4a3f-8fc1-61c702b89f63\") " pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472777 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-stats-auth\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472808 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6eb01e1-380a-468c-8097-d34540808454-env-overrides\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472839 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88f82f59-91bd-4845-ae6b-ca143189342d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472901 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca92ce4d-7cf4-4da5-8609-920db260c2d3-original-pull-secret\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472916 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-default-certificate\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472905 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-config-volume\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.472926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472904 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-multus-daemon-config\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.473317 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.472904 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/580a0715-a63d-43ef-983c-6d459af6ba42-cni-binary-copy\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.474031 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.473992 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dbc159e-0969-417d-91fb-819d526579a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.474099 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.474047 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/271984b8-d201-41a3-a545-690f8a9fad89-serving-cert\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:59:01.474099 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.474090 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/271984b8-d201-41a3-a545-690f8a9fad89-config\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:59:01.474177 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.474137 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-iptables-alerter-script\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:59:01.513568 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.513541 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2kh4\" (UniqueName: \"kubernetes.io/projected/362105ce-9f40-4ad3-964b-472512bf9518-kube-api-access-n2kh4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz\" (UID: \"362105ce-9f40-4ad3-964b-472512bf9518\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" Nov 03 06:59:01.513701 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.513553 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqd2\" (UniqueName: \"kubernetes.io/projected/40c1e3ef-149c-401f-8e88-8bdafa0faae6-kube-api-access-blqd2\") pod \"kas-connection-checker-685895c7c9-hzqt4\" (UID: \"40c1e3ef-149c-401f-8e88-8bdafa0faae6\") " pod="kube-system/kas-connection-checker-685895c7c9-hzqt4" Nov 03 06:59:01.514159 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514066 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh76m\" (UniqueName: \"kubernetes.io/projected/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-kube-api-access-bh76m\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:59:01.514159 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514143 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.514382 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514176 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzkl\" (UniqueName: \"kubernetes.io/projected/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-kube-api-access-pdzkl\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.514382 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514212 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhbg\" (UniqueName: \"kubernetes.io/projected/1bfb1d47-8ee3-4e09-a4ea-ddcef2244233-kube-api-access-gnhbg\") pod \"migrator-66df5d496b-29ktd\" (UID: \"1bfb1d47-8ee3-4e09-a4ea-ddcef2244233\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" Nov 03 06:59:01.514382 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514340 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch4qf\" (UniqueName: \"kubernetes.io/projected/8346ef93-89cc-47f7-b3bf-3a309b6efa2d-kube-api-access-ch4qf\") pod \"ingress-canary-fstk4\" (UID: \"8346ef93-89cc-47f7-b3bf-3a309b6efa2d\") " pod="openshift-ingress-canary/ingress-canary-fstk4" Nov 03 06:59:01.514382 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514359 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7c5q\" (UniqueName: \"kubernetes.io/projected/271984b8-d201-41a3-a545-690f8a9fad89-kube-api-access-q7c5q\") pod \"service-ca-operator-544d749c8c-4n4t6\" (UID: \"271984b8-d201-41a3-a545-690f8a9fad89\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" Nov 03 06:59:01.514636 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514545 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gp8\" (UniqueName: \"kubernetes.io/projected/66c763d5-1fbd-428a-bf62-cb4102fbc145-kube-api-access-g5gp8\") pod \"volume-data-source-validator-58c9f5f46d-z4nr5\" (UID: \"66c763d5-1fbd-428a-bf62-cb4102fbc145\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z4nr5" Nov 03 06:59:01.514636 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514603 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9m9\" (UniqueName: \"kubernetes.io/projected/e6eb01e1-380a-468c-8097-d34540808454-kube-api-access-tc9m9\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.514636 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514609 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffvz\" (UniqueName: \"kubernetes.io/projected/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-kube-api-access-sffvz\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:59:01.514892 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514871 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4f7r\" (UniqueName: \"kubernetes.io/projected/95020920-a029-4c17-8ce1-e66a4982a00d-kube-api-access-p4f7r\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.514926 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.514902 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjv9\" (UniqueName: \"kubernetes.io/projected/0dbc159e-0969-417d-91fb-819d526579a3-kube-api-access-qtjv9\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.515181 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.515164 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2xn\" (UniqueName: \"kubernetes.io/projected/452ab275-c3b0-4c31-8603-7af7cd666346-kube-api-access-2k2xn\") pod \"console-operator-7cd56b8dbc-ldktd\" (UID: \"452ab275-c3b0-4c31-8603-7af7cd666346\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:01.518522 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.518503 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrjh6\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.518630 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.518550 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.518819 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.518802 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qpj\" (UniqueName: \"kubernetes.io/projected/25276ee9-63e0-4d32-834e-8223c741b724-kube-api-access-b9qpj\") pod \"kas-connection-checker-685895c7c9-5ndhb\" (UID: \"25276ee9-63e0-4d32-834e-8223c741b724\") " pod="kube-system/kas-connection-checker-685895c7c9-5ndhb" Nov 03 06:59:01.519082 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.519033 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqdw\" (UniqueName: \"kubernetes.io/projected/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kube-api-access-ckqdw\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.519341 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.519323 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2vm\" (UniqueName: \"kubernetes.io/projected/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-kube-api-access-cv2vm\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.519596 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.519573 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfg9\" (UniqueName: \"kubernetes.io/projected/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-api-access-tnfg9\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.519715 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.519597 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9wc\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.519715 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.519576 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2xf\" (UniqueName: \"kubernetes.io/projected/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-kube-api-access-cj2xf\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:01.519887 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.519869 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpp4t\" (UniqueName: \"kubernetes.io/projected/4e7d1d1e-2fb6-412d-8597-f6f4882d0389-kube-api-access-cpp4t\") pod \"kas-connection-checker-685895c7c9-4jk57\" (UID: \"4e7d1d1e-2fb6-412d-8597-f6f4882d0389\") " pod="kube-system/kas-connection-checker-685895c7c9-4jk57" Nov 03 06:59:01.520083 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.520063 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxnh\" (UniqueName: \"kubernetes.io/projected/bee72bc7-52f6-45a6-8118-eefba66e6992-kube-api-access-rdxnh\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:01.520136 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.520094 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfgw\" (UniqueName: \"kubernetes.io/projected/580a0715-a63d-43ef-983c-6d459af6ba42-kube-api-access-wxfgw\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.520220 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.520205 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc428\" (UniqueName: \"kubernetes.io/projected/e47cc873-c3fe-421e-8882-9ce4b4ff3737-kube-api-access-gc428\") pod \"network-check-target-dfpc7\" (UID: \"e47cc873-c3fe-421e-8882-9ce4b4ff3737\") " pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:59:01.520515 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.520499 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc487\" (UniqueName: \"kubernetes.io/projected/902e5dbc-172c-496e-afaf-2e847ba1f852-kube-api-access-pc487\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:01.521170 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.521155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glrf\" (UniqueName: \"kubernetes.io/projected/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-kube-api-access-2glrf\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:59:01.524414 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.524398 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qscpq\" (UniqueName: \"kubernetes.io/projected/728637b6-bbb0-4532-adde-4555c4f4153d-kube-api-access-qscpq\") pod \"insights-operator-7c4986d449-299kl\" (UID: \"728637b6-bbb0-4532-adde-4555c4f4153d\") " pod="openshift-insights/insights-operator-7c4986d449-299kl" Nov 03 06:59:01.524572 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.524553 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pw6\" (UniqueName: \"kubernetes.io/projected/936c5f59-3e71-4d99-ad3a-b0762bf8afda-kube-api-access-z2pw6\") pod \"network-check-source-6f78d59f99-fb26v\" (UID: \"936c5f59-3e71-4d99-ad3a-b0762bf8afda\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fb26v" Nov 03 06:59:01.555632 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555608 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-modprobe-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.555698 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555639 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-kubelet\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.555698 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555657 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555698 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555672 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-bin\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555698 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555687 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysconfig\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555706 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555720 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555718 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-kubelet\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555762 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555765 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-bin\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555774 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysconfig\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555785 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-modprobe-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555803 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-conf-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555840 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-ovn\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555841 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-conf-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555865 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-ovn\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555865 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-hosts-file\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:59:01.555893 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555894 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-var-lib-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555920 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-kubelet\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555943 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-kubernetes\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555959 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-var-lib-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555963 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eab2a8b5-c6f2-45df-9fcc-161d6ad92498-hosts-file\") pod \"node-resolver-zq5qq\" (UID: \"eab2a8b5-c6f2-45df-9fcc-161d6ad92498\") " pod="openshift-dns/node-resolver-zq5qq" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555972 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-kubelet\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555967 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.555999 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-kubernetes\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556068 4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556071 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556078 4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556116 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-sys\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556140 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556145 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-k8s-cni-cncf-io\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556158 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-sys\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556167 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-host\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556186 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-k8s-cni-cncf-io\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556195 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.056175286 +0000 UTC m=+1.989703943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556206 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-host\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556214 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-cnibin\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556218 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.056208041 +0000 UTC m=+1.989736700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556235 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-cnibin\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556246 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556264 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556278 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/018a989a-6542-4db1-bd2a-d80df7bbed24-crio-socket\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556293 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.556306 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556307 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-hostroot\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556323 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556344 4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556346 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-dbus\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556355 4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556365 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/018a989a-6542-4db1-bd2a-d80df7bbed24-crio-socket\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556371 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-registration-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556379 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-hostroot\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556399 4925 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556401 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.056389582 +0000 UTC m=+1.989918231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556421 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.056411538 +0000 UTC m=+1.989940174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556433 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556472 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-system-cni-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556485 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-dbus\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556491 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.05647603 +0000 UTC m=+1.990004682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556514 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-system-cni-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556520 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.05651006 +0000 UTC m=+1.990038700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : configmap "kube-rbac-proxy" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556515 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-registration-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556544 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556574 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556591 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556600 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-socket-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556618 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556642 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.056627076 +0000 UTC m=+1.990155728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556649 4925 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556665 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-system-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556677 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbc159e-0969-417d-91fb-819d526579a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5cxd\" (UID: \"0dbc159e-0969-417d-91fb-819d526579a3\") " pod="openshift-multus/multus-additional-cni-plugins-x5cxd" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556693 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.556704 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.05669225 +0000 UTC m=+1.990220903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556704 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-socket-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556732 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-system-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556738 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-netns\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556762 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-run-netns\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556771 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-var-lib-kubelet\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556738 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-cni-dir\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556795 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-device-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556822 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-systemd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556842 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-var-lib-kubelet\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556852 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-systemd\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556866 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-device-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556880 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-systemd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556880 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-systemd-units\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556898 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-systemd\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556915 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-systemd-units\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556922 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-etc-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556951 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-cnibin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556966 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-netns\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556972 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-etc-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556983 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-etc-kubernetes\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.556999 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557002 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-netns\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557001 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-cnibin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557036 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557036 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-etc-kubernetes\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557049 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-run-openvswitch\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557064 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-kubelet-config\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557086 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-lib-modules\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557102 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-sys-fs\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557106 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca92ce4d-7cf4-4da5-8609-920db260c2d3-kubelet-config\") pod \"global-pull-secret-syncer-dtzxb\" (UID: \"ca92ce4d-7cf4-4da5-8609-920db260c2d3\") " pod="kube-system/global-pull-secret-syncer-dtzxb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557113 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-d\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557122 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-bin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557141 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-conf\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557155 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-run\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-sys-fs\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557168 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-host-slash\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557181 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-lib-modules\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557185 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-multus\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557212 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-netd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557221 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-bin\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557224 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-run\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557229 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b39d84-7483-46fb-ab4b-d8de4e3a1212-host-slash\") pod \"iptables-alerter-vn65v\" (UID: \"c6b39d84-7483-46fb-ab4b-d8de4e3a1212\") " pod="openshift-network-operator/iptables-alerter-vn65v" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557233 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557260 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-cni-netd\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557263 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-socket-dir-parent\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557215 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-var-lib-cni-multus\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557294 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95020920-a029-4c17-8ce1-e66a4982a00d-etc-sysctl-conf\") pod \"tuned-zhjnj\" (UID: \"95020920-a029-4c17-8ce1-e66a4982a00d\") " pod="openshift-cluster-node-tuning-operator/tuned-zhjnj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557291 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557304 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557319 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-multus-certs\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557335 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-slash\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.557351 4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557381 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-host-slash\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557350 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.557399 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557419 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557318 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-multus-socket-dir-parent\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557349 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/580a0715-a63d-43ef-983c-6d459af6ba42-host-run-multus-certs\") pod \"multus-v9wll\" (UID: \"580a0715-a63d-43ef-983c-6d459af6ba42\") " pod="openshift-multus/multus-v9wll" Nov 03 06:59:01.557393 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.557429 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.057417652 +0000 UTC m=+1.990946323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.557465 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.057455551 +0000 UTC m=+1.990984185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557472 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e402a1-d12e-4559-aae4-38ca72e7f1cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dlwnb\" (UID: \"10e402a1-d12e-4559-aae4-38ca72e7f1cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dlwnb" Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557480 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-node-log\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557507 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557532 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-host\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557556 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-log-socket\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.557565 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557509 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-node-log\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:01.557601 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:02.057588263 +0000 UTC m=+1.991116931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : secret "metrics-daemon-secret" not found Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557621 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6eb01e1-380a-468c-8097-d34540808454-log-socket\") pod \"ovnkube-node-pb8jn\" (UID: \"e6eb01e1-380a-468c-8097-d34540808454\") " pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:01.559749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.557620 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fc2a19a-80f8-4318-a6fd-b246b3022d0e-host\") pod \"node-ca-9vk5q\" (UID: \"2fc2a19a-80f8-4318-a6fd-b246b3022d0e\") " pod="openshift-image-registry/node-ca-9vk5q" Nov 03 06:59:01.610280 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.610250 4925 scope.go:122] "RemoveContainer" containerID="ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3" Nov 03 06:59:01.624743 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:01.624725 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:59:02.062403 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062324 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:02.062403 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062361 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:02.062403 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062378 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062458 4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062468 4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062479 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062463145 +0000 UTC m=+2.995991804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062504 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062494326 +0000 UTC m=+2.996022960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062506 4925 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062516 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062538 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062546 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062536564 +0000 UTC m=+2.996065201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : configmap "kube-rbac-proxy" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062579 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062601 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.06259462 +0000 UTC m=+2.996123254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062573 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062623 4925 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062629 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062657 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062661 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062653551 +0000 UTC m=+2.996182219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062681 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062671817 +0000 UTC m=+2.996200456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062688 4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062702 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062706 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062700178 +0000 UTC m=+2.996228812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062730 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062738 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:02.062756 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062762 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062752447 +0000 UTC m=+2.996281082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062780 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062786 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062797 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062791917 +0000 UTC m=+2.996320551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : secret "metrics-daemon-secret" not found Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.062832 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062862 4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062876 4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062895 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062911 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062901838 +0000 UTC m=+2.996430476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:59:02.063480 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.062922 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.062917063 +0000 UTC m=+2.996445703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:59:02.229747 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.229708 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:02.244441 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.244417 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 06:59:02.244877 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.244861 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/1.log" Nov 03 06:59:02.244950 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.244893 4925 generic.go:356] "Generic (PLEG): container finished" podID="452ab275-c3b0-4c31-8603-7af7cd666346" containerID="7eeb5f267af2fd906bb60ba4ba8ae0ee92640babf617a84448345f023264cf21" exitCode=255 Nov 03 06:59:02.244994 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.244975 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" event={"ID":"452ab275-c3b0-4c31-8603-7af7cd666346","Type":"ContainerDied","Data":"7eeb5f267af2fd906bb60ba4ba8ae0ee92640babf617a84448345f023264cf21"} Nov 03 06:59:02.245078 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.245065 4925 scope.go:122] "RemoveContainer" containerID="ea48035110f076222ed114a622abaed0534b197cf36dbc42e55350fc5945ead3" Nov 03 06:59:02.245345 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:02.245326 4925 scope.go:122] "RemoveContainer" containerID="7eeb5f267af2fd906bb60ba4ba8ae0ee92640babf617a84448345f023264cf21" Nov 03 06:59:02.245534 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:02.245515 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ldktd_openshift-console-operator(452ab275-c3b0-4c31-8603-7af7cd666346)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" podUID="452ab275-c3b0-4c31-8603-7af7cd666346" Nov 03 06:59:03.072436 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.072390 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:03.072645 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.072577 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:03.072714 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.072653 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.072624231 +0000 UTC m=+5.006152871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : secret "metrics-daemon-secret" not found Nov 03 06:59:03.072802 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.072788 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:03.072863 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.072837 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.072824917 +0000 UTC m=+5.006353567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:59:03.072928 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.072455 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:03.073050 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073031 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:03.073119 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073080 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:03.073171 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073132 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:03.073171 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073163 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:03.073259 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073195 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:03.073259 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073234 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:03.073352 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073282 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:03.073352 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073314 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:03.073436 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.073401 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:03.073573 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073552 4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:59:03.073630 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073602 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.073589878 +0000 UTC m=+5.007118512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:59:03.073710 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073699 4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:59:03.073755 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073714 4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57998bb766-8gfrg: secret "image-registry-tls" not found Nov 03 06:59:03.073755 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073752 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls podName:063967c5-741c-46f1-a023-9384b59ccaa7 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.073741308 +0000 UTC m=+5.007269946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls") pod "image-registry-57998bb766-8gfrg" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7") : secret "image-registry-tls" not found Nov 03 06:59:03.073862 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073850 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:59:03.073911 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073892 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.073881781 +0000 UTC m=+5.007410417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:59:03.073989 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.073978 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.073961837 +0000 UTC m=+5.007490474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : configmap references non-existent config key: service-ca.crt Nov 03 06:59:03.074082 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074070 4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 06:59:03.074130 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074085 4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54fc5dcf5f-l5q5j: secret "image-registry-tls" not found Nov 03 06:59:03.074130 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074122 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls podName:f55630a1-1afe-446c-bda0-20bfba394fb5 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.074111583 +0000 UTC m=+5.007640231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls") pod "image-registry-54fc5dcf5f-l5q5j" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5") : secret "image-registry-tls" not found Nov 03 06:59:03.074233 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074213 4925 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:59:03.074278 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074268 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.074238978 +0000 UTC m=+5.007767625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : configmap "kube-rbac-proxy" not found Nov 03 06:59:03.075997 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074339 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:03.075997 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074388 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.074362079 +0000 UTC m=+5.007890732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:03.075997 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074459 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:03.075997 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074495 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.074479025 +0000 UTC m=+5.008007685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:03.075997 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074572 4925 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:59:03.075997 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:03.074611 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.074595934 +0000 UTC m=+5.008124582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:59:03.249484 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.249454 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-29ktd" event={"ID":"1bfb1d47-8ee3-4e09-a4ea-ddcef2244233","Type":"ContainerStarted","Data":"23656ecd943dd4f5991355125a8c33e1f1d91ecbd6aff3f97346ac655d91d4b7"} Nov 03 06:59:03.251065 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.251046 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 06:59:03.320226 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.320196 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:03.569663 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.569636 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:03.795327 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.795292 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:03.995316 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:03.995193 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:04.196623 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:04.196590 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:04.448921 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:04.448883 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:04.647190 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:04.647158 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:04.858773 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:04.858695 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:05.063741 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.063709 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:05.090820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090800 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:05.090947 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090830 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:05.090947 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090852 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:05.090947 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090870 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:05.090947 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090909 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:05.090947 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090927 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:05.090947 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090945 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.090957 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.090983 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.091031 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091035 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091043 4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091045 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls podName:902e5dbc-172c-496e-afaf-2e847ba1f852 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.091005659 +0000 UTC m=+9.024534292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n6xht" (UID: "902e5dbc-172c-496e-afaf-2e847ba1f852") : secret "cluster-monitoring-operator-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091090 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091090 4925 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091097 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091112 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.091096222 +0000 UTC m=+9.024624856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091133 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs podName:84d699e8-7ccc-4e2c-b6c6-e309946bee2e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.091121135 +0000 UTC m=+9.024649773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs") pod "router-default-699757bbb7-drfpj" (UID: "84d699e8-7ccc-4e2c-b6c6-e309946bee2e") : secret "router-metrics-certs-default" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091151 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091158 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.091143202 +0000 UTC m=+9.024671838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091177 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert podName:88f82f59-91bd-4845-ae6b-ca143189342d nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.091168727 +0000 UTC m=+9.024697362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpk6z" (UID: "88f82f59-91bd-4845-ae6b-ca143189342d") : secret "networking-console-plugin-cert" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091195 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.091187066 +0000 UTC m=+9.024715703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.091213 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.0912038 +0000 UTC m=+9.024732434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : secret "metrics-daemon-secret" not found Nov 03 06:59:05.091256 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.091248 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:05.091781 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.091278 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:05.091781 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.091416 4925 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 06:59:05.091781 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.091487 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/018a989a-6542-4db1-bd2a-d80df7bbed24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:05.091880 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.091838 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-service-ca-bundle\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:05.093739 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.093717 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"image-registry-57998bb766-8gfrg\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:05.093831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.093812 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"image-registry-54fc5dcf5f-l5q5j\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:05.208619 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.208561 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:05.219948 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.219930 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:05.457847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.457821 4925 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:05.458236 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.458149 4925 scope.go:122] "RemoveContainer" containerID="7eeb5f267af2fd906bb60ba4ba8ae0ee92640babf617a84448345f023264cf21" Nov 03 06:59:05.458348 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:05.458330 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ldktd_openshift-console-operator(452ab275-c3b0-4c31-8603-7af7cd666346)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" podUID="452ab275-c3b0-4c31-8603-7af7cd666346" Nov 03 06:59:05.651554 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.651524 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:05.707139 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.707112 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54fc5dcf5f-l5q5j"] Nov 03 06:59:05.709876 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.709857 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57998bb766-8gfrg"] Nov 03 06:59:05.712264 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:05.712230 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063967c5_741c_46f1_a023_9384b59ccaa7.slice/crio-27d2a46b9ce4f48d9fadf25171125e8da044a157536ffaba7292db0a0b58ef3f WatchSource:0}: Error finding container 27d2a46b9ce4f48d9fadf25171125e8da044a157536ffaba7292db0a0b58ef3f: Status 404 returned error can't find the container with id 27d2a46b9ce4f48d9fadf25171125e8da044a157536ffaba7292db0a0b58ef3f Nov 03 06:59:05.758367 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:05.758348 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55630a1_1afe_446c_bda0_20bfba394fb5.slice/crio-7a9a38e9265dc7076087c01009c66c6261abbb12576fc1e59bc205ca4b9e0653 WatchSource:0}: Error finding container 7a9a38e9265dc7076087c01009c66c6261abbb12576fc1e59bc205ca4b9e0653: Status 404 returned error can't find the container with id 7a9a38e9265dc7076087c01009c66c6261abbb12576fc1e59bc205ca4b9e0653 Nov 03 06:59:05.876984 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:05.876948 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:06.079596 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:06.079544 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:06.270535 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:06.270502 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" event={"ID":"063967c5-741c-46f1-a023-9384b59ccaa7","Type":"ContainerStarted","Data":"d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0"} Nov 03 06:59:06.270710 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:06.270554 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" event={"ID":"063967c5-741c-46f1-a023-9384b59ccaa7","Type":"ContainerStarted","Data":"27d2a46b9ce4f48d9fadf25171125e8da044a157536ffaba7292db0a0b58ef3f"} Nov 03 06:59:06.271983 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:06.271960 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" event={"ID":"f55630a1-1afe-446c-bda0-20bfba394fb5","Type":"ContainerStarted","Data":"fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b"} Nov 03 06:59:06.272114 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:06.271997 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" event={"ID":"f55630a1-1afe-446c-bda0-20bfba394fb5","Type":"ContainerStarted","Data":"7a9a38e9265dc7076087c01009c66c6261abbb12576fc1e59bc205ca4b9e0653"} Nov 03 06:59:06.461488 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:06.461417 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:06.614000 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:06.613962 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:59:07.083914 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.083879 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:07.275746 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.275718 4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 06:59:07.275919 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.275718 4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 06:59:07.428060 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.427979 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zq5qq_eab2a8b5-c6f2-45df-9fcc-161d6ad92498/dns-node-resolver/0.log" Nov 03 06:59:07.628211 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.628181 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54fc5dcf5f-l5q5j_f55630a1-1afe-446c-bda0-20bfba394fb5/registry/0.log" Nov 03 06:59:07.664620 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.664594 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:07.828831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.828803 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57998bb766-8gfrg_063967c5-741c-46f1-a023-9384b59ccaa7/registry/0.log" Nov 03 06:59:07.904843 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:07.904819 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:08.031400 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:08.031374 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9vk5q_2fc2a19a-80f8-4318-a6fd-b246b3022d0e/node-ca/0.log" Nov 03 06:59:08.136180 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:08.136107 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:08.136502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:08.136489 4925 scope.go:122] "RemoveContainer" containerID="7eeb5f267af2fd906bb60ba4ba8ae0ee92640babf617a84448345f023264cf21" Nov 03 06:59:08.136697 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:08.136676 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ldktd_openshift-console-operator(452ab275-c3b0-4c31-8603-7af7cd666346)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" podUID="452ab275-c3b0-4c31-8603-7af7cd666346" Nov 03 06:59:08.258107 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:08.258072 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:08.510528 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:08.510445 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:08.532329 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:08.532305 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:08.967343 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:08.967318 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:09.027545 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.027522 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fstk4_8346ef93-89cc-47f7-b3bf-3a309b6efa2d/serve-healthcheck-canary/0.log" Nov 03 06:59:09.085427 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.085395 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:09.130530 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.130506 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:09.130660 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.130554 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:09.130660 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.130589 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:09.130660 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.130614 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:09.130801 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.130655 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 06:59:09.130801 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.130708 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls podName:bee72bc7-52f6-45a6-8118-eefba66e6992 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:17.130693664 +0000 UTC m=+17.064222314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h9vl4" (UID: "bee72bc7-52f6-45a6-8118-eefba66e6992") : secret "samples-operator-tls" not found Nov 03 06:59:09.130801 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.130730 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:09.130801 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.130793 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls podName:018a989a-6542-4db1-bd2a-d80df7bbed24 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:17.130776199 +0000 UTC m=+17.064304852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jjq57" (UID: "018a989a-6542-4db1-bd2a-d80df7bbed24") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:09.130985 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.130830 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:09.130985 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.130865 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:09.130985 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.130895 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:09.131183 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.130983 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:09.131183 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.131000 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:09.131183 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.131059 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls podName:e15cc64f-4c50-4c6d-89fd-d2144ad0d38c nodeName:}" failed. No retries permitted until 2026-11-03 06:59:17.131045445 +0000 UTC m=+17.064574082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls") pod "dns-default-cpnw7" (UID: "e15cc64f-4c50-4c6d-89fd-d2144ad0d38c") : secret "dns-default-metrics-tls" not found Nov 03 06:59:09.131183 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:09.131074 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs podName:c9627c15-ddb9-4ca1-b1d1-606c8c0453b3 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:17.131068289 +0000 UTC m=+17.064596922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs") pod "network-metrics-daemon-g8m65" (UID: "c9627c15-ddb9-4ca1-b1d1-606c8c0453b3") : secret "metrics-daemon-secret" not found Nov 03 06:59:09.133155 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.133132 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88f82f59-91bd-4845-ae6b-ca143189342d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpk6z\" (UID: \"88f82f59-91bd-4845-ae6b-ca143189342d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:09.133155 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.133144 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84d699e8-7ccc-4e2c-b6c6-e309946bee2e-metrics-certs\") pod \"router-default-699757bbb7-drfpj\" (UID: \"84d699e8-7ccc-4e2c-b6c6-e309946bee2e\") " pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:09.133155 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.133149 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/902e5dbc-172c-496e-afaf-2e847ba1f852-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n6xht\" (UID: \"902e5dbc-172c-496e-afaf-2e847ba1f852\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:09.281925 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.281852 4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 06:59:09.413369 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.413340 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" Nov 03 06:59:09.414460 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.414432 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:09.415486 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.415386 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" Nov 03 06:59:09.428954 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.428914 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-29ktd_1bfb1d47-8ee3-4e09-a4ea-ddcef2244233/migrator/0.log" Nov 03 06:59:09.698334 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.698307 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:09.829260 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.829227 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz_362105ce-9f40-4ad3-964b-472512bf9518/kube-storage-version-migrator-operator/0.log" Nov 03 06:59:09.848419 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.848397 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht"] Nov 03 06:59:09.851223 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.851201 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lpk6z"] Nov 03 06:59:09.851328 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:09.851276 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902e5dbc_172c_496e_afaf_2e847ba1f852.slice/crio-eff5208a84ea6388345a0760509aa772cd6fb4b9f214bc24bd4b47f958565125 WatchSource:0}: Error finding container eff5208a84ea6388345a0760509aa772cd6fb4b9f214bc24bd4b47f958565125: Status 404 returned error can't find the container with id eff5208a84ea6388345a0760509aa772cd6fb4b9f214bc24bd4b47f958565125 Nov 03 06:59:09.887564 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:09.887541 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f82f59_91bd_4845_ae6b_ca143189342d.slice/crio-40601c8e7c0ecd779e71a5348480291653c344e4ef8ebe18d0a78cc7efc758a5 WatchSource:0}: Error finding container 40601c8e7c0ecd779e71a5348480291653c344e4ef8ebe18d0a78cc7efc758a5: Status 404 returned error can't find the container with id 40601c8e7c0ecd779e71a5348480291653c344e4ef8ebe18d0a78cc7efc758a5 Nov 03 06:59:09.903727 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.903708 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-699757bbb7-drfpj"] Nov 03 06:59:09.906053 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:09.906030 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d699e8_7ccc_4e2c_b6c6_e309946bee2e.slice/crio-6773fefa040a30abd1c3d05444dd3ca8e6b0011683b93f936580a6730d11ebe5 WatchSource:0}: Error finding container 6773fefa040a30abd1c3d05444dd3ca8e6b0011683b93f936580a6730d11ebe5: Status 404 returned error can't find the container with id 6773fefa040a30abd1c3d05444dd3ca8e6b0011683b93f936580a6730d11ebe5 Nov 03 06:59:09.940862 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:09.940831 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:10.205908 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.205877 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:10.285809 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.285775 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-699757bbb7-drfpj" event={"ID":"84d699e8-7ccc-4e2c-b6c6-e309946bee2e","Type":"ContainerStarted","Data":"db8c87b86012c56bc25ad03ed65bde96170f85bf37ff6baced2a77a8f4788d93"} Nov 03 06:59:10.285971 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.285830 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-699757bbb7-drfpj" event={"ID":"84d699e8-7ccc-4e2c-b6c6-e309946bee2e","Type":"ContainerStarted","Data":"6773fefa040a30abd1c3d05444dd3ca8e6b0011683b93f936580a6730d11ebe5"} Nov 03 06:59:10.286927 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.286896 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" event={"ID":"902e5dbc-172c-496e-afaf-2e847ba1f852","Type":"ContainerStarted","Data":"eff5208a84ea6388345a0760509aa772cd6fb4b9f214bc24bd4b47f958565125"} Nov 03 06:59:10.288029 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.287978 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" event={"ID":"88f82f59-91bd-4845-ae6b-ca143189342d","Type":"ContainerStarted","Data":"40601c8e7c0ecd779e71a5348480291653c344e4ef8ebe18d0a78cc7efc758a5"} Nov 03 06:59:10.305202 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.305178 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:10.414649 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.414616 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:10.417211 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.417182 4925 patch_prober.go:29] interesting pod/router-default-699757bbb7-drfpj container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 03 06:59:10.417211 ip-10-0-136-184 kubenswrapper[4925]: [+]has-synced ok Nov 03 06:59:10.417211 ip-10-0-136-184 kubenswrapper[4925]: [+]process-running ok Nov 03 06:59:10.417211 ip-10-0-136-184 kubenswrapper[4925]: healthz check failed Nov 03 06:59:10.417401 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.417242 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-699757bbb7-drfpj" podUID="84d699e8-7ccc-4e2c-b6c6-e309946bee2e" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 03 06:59:10.548258 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.548188 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:10.633327 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.633300 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pb8jn" Nov 03 06:59:10.998195 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:10.998158 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:11.212330 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:11.212296 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:11.416904 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:11.416875 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:11.682398 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:11.682296 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:11.684808 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:11.684787 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4gwnd" Nov 03 06:59:11.704408 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:11.704384 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:11.706319 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:11.706296 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-699757bbb7-drfpj" Nov 03 06:59:11.961309 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:11.961280 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:12.264102 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:12.264001 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:12.296609 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:12.296583 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n6xht" event={"ID":"902e5dbc-172c-496e-afaf-2e847ba1f852","Type":"ContainerStarted","Data":"0f3fec9f6e424802f180cd0a0ec212e4381424b2887cf07da0ced9251d92b08d"} Nov 03 06:59:12.298061 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:12.298036 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpk6z" event={"ID":"88f82f59-91bd-4845-ae6b-ca143189342d","Type":"ContainerStarted","Data":"825e33282f3dcd1df114d10fb354f13a5c0d766f8d480cd8fbf7c44b611b4e2e"} Nov 03 06:59:12.477923 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:12.477892 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:12.527218 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:12.527149 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:13.062500 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.062461 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:13.184203 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.184177 4925 scope.go:122] "RemoveContainer" containerID="7eeb5f267af2fd906bb60ba4ba8ae0ee92640babf617a84448345f023264cf21" Nov 03 06:59:13.275420 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.275393 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:13.303289 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.303270 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 06:59:13.303421 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.303397 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" event={"ID":"452ab275-c3b0-4c31-8603-7af7cd666346","Type":"ContainerStarted","Data":"f2f982c2cf4144d3e440596c638dc9f18b1d48ce8f623c297e349a3c960a161e"} Nov 03 06:59:13.303990 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.303970 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:13.304721 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.304702 4925 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-ldktd container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Nov 03 06:59:13.304810 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.304744 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" podUID="452ab275-c3b0-4c31-8603-7af7cd666346" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Nov 03 06:59:13.519418 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.519383 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:13.735566 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.735528 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:13.944965 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:13.944931 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:14.168884 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:14.168846 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:14.311168 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:14.311097 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ldktd" Nov 03 06:59:14.367316 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:14.367289 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:14.579946 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:14.579916 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:14.812644 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:14.812598 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:15.057143 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:15.057063 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:15.275022 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:15.274972 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:15.507792 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:15.507756 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:15.735611 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:15.735570 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:16.156347 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:16.156314 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:16.366519 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:16.366476 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:17.195811 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.195775 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:17.196320 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.195836 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:17.196320 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.195857 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:17.196320 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.195899 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:17.198544 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.198516 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/018a989a-6542-4db1-bd2a-d80df7bbed24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jjq57\" (UID: \"018a989a-6542-4db1-bd2a-d80df7bbed24\") " pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:17.198676 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.198563 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9627c15-ddb9-4ca1-b1d1-606c8c0453b3-metrics-certs\") pod \"network-metrics-daemon-g8m65\" (UID: \"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3\") " pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:17.198676 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.198565 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e15cc64f-4c50-4c6d-89fd-d2144ad0d38c-metrics-tls\") pod \"dns-default-cpnw7\" (UID: \"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c\") " pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:17.198676 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.198639 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee72bc7-52f6-45a6-8118-eefba66e6992-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h9vl4\" (UID: \"bee72bc7-52f6-45a6-8118-eefba66e6992\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:17.214491 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.214471 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" Nov 03 06:59:17.218256 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.218237 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8m65" Nov 03 06:59:17.222889 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.222873 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jjq57" Nov 03 06:59:17.222979 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.222961 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:17.604923 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.604893 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g8m65"] Nov 03 06:59:17.643780 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:17.643756 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9627c15_ddb9_4ca1_b1d1_606c8c0453b3.slice/crio-8f206601907ae018e810bdd8b25480aefce86944d740c06d77074d2244456fc7 WatchSource:0}: Error finding container 8f206601907ae018e810bdd8b25480aefce86944d740c06d77074d2244456fc7: Status 404 returned error can't find the container with id 8f206601907ae018e810bdd8b25480aefce86944d740c06d77074d2244456fc7 Nov 03 06:59:17.656128 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.656109 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4"] Nov 03 06:59:17.692905 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:17.692882 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15cc64f_4c50_4c6d_89fd_d2144ad0d38c.slice/crio-d0c3f03fee3cc1ec1fdfc973b81608293429d29d288c9ee590ca5c168baff855 WatchSource:0}: Error finding container d0c3f03fee3cc1ec1fdfc973b81608293429d29d288c9ee590ca5c168baff855: Status 404 returned error can't find the container with id d0c3f03fee3cc1ec1fdfc973b81608293429d29d288c9ee590ca5c168baff855 Nov 03 06:59:17.707318 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.707298 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cpnw7"] Nov 03 06:59:17.709154 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:17.709127 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jjq57"] Nov 03 06:59:17.711301 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:17.711278 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018a989a_6542_4db1_bd2a_d80df7bbed24.slice/crio-62b54be5768a806c2417359fef382a9be921de8ead40098a7b36de8648c903e9 WatchSource:0}: Error finding container 62b54be5768a806c2417359fef382a9be921de8ead40098a7b36de8648c903e9: Status 404 returned error can't find the container with id 62b54be5768a806c2417359fef382a9be921de8ead40098a7b36de8648c903e9 Nov 03 06:59:18.339447 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:18.339414 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpnw7" event={"ID":"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c","Type":"ContainerStarted","Data":"d0c3f03fee3cc1ec1fdfc973b81608293429d29d288c9ee590ca5c168baff855"} Nov 03 06:59:18.340905 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:18.340883 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jjq57" event={"ID":"018a989a-6542-4db1-bd2a-d80df7bbed24","Type":"ContainerStarted","Data":"54a2f38333c498251f933e810a29eade579d481e22b756c8702d2e976cfd5d98"} Nov 03 06:59:18.341035 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:18.340935 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jjq57" event={"ID":"018a989a-6542-4db1-bd2a-d80df7bbed24","Type":"ContainerStarted","Data":"62b54be5768a806c2417359fef382a9be921de8ead40098a7b36de8648c903e9"} Nov 03 06:59:18.342050 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:18.342006 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8m65" event={"ID":"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3","Type":"ContainerStarted","Data":"8f206601907ae018e810bdd8b25480aefce86944d740c06d77074d2244456fc7"} Nov 03 06:59:18.343037 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:18.343005 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" event={"ID":"bee72bc7-52f6-45a6-8118-eefba66e6992","Type":"ContainerStarted","Data":"8854bc1206bb880971802d6077b474473205c65101b86831f0e2ea35c045dda4"} Nov 03 06:59:19.349587 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:19.349541 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jjq57" event={"ID":"018a989a-6542-4db1-bd2a-d80df7bbed24","Type":"ContainerStarted","Data":"9c20429b03f73dd2392fbca87ec3217ec4e60a6fcefd4bbd057d80cbe9276ea0"} Nov 03 06:59:20.211303 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:20.211273 4925 patch_prober.go:29] interesting pod/image-registry-57998bb766-8gfrg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 06:59:20.211421 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:20.211324 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" podUID="063967c5-741c-46f1-a023-9384b59ccaa7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:59:20.356446 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:20.356419 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpnw7" event={"ID":"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c","Type":"ContainerStarted","Data":"76cf2525a3dc58ea909dd207c0bcc1452efbbdfd340a30cbc339b3883fa8edab"} Nov 03 06:59:20.359232 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:20.359208 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8m65" event={"ID":"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3","Type":"ContainerStarted","Data":"2342d3f8b5121e0b3d604e0d3afd51b00f4608362c9827e8d37f637229696533"} Nov 03 06:59:20.359336 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:20.359260 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8m65" event={"ID":"c9627c15-ddb9-4ca1-b1d1-606c8c0453b3","Type":"ContainerStarted","Data":"43cd4599b9fbdde94f3b6e160705a761dc93057bb1896042a17e50dcc0f04efa"} Nov 03 06:59:20.363855 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:20.363832 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" event={"ID":"bee72bc7-52f6-45a6-8118-eefba66e6992","Type":"ContainerStarted","Data":"758b11b84336614de40c924be22b186aea331ea99f37c922544c4bb704683d91"} Nov 03 06:59:21.369689 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.369657 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpnw7" event={"ID":"e15cc64f-4c50-4c6d-89fd-d2144ad0d38c","Type":"ContainerStarted","Data":"d963baacb6c786e5a483c5e717f49f896d5a986efa3ec65fee12c1f8cf0d308c"} Nov 03 06:59:21.370116 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.369773 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:21.371987 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.371967 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jjq57" event={"ID":"018a989a-6542-4db1-bd2a-d80df7bbed24","Type":"ContainerStarted","Data":"42d8a5b90593eabd81a65ec05ef3d0704ee2e7c00ab2a88788fe8eb3e1d7643e"} Nov 03 06:59:21.373681 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.373656 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h9vl4" event={"ID":"bee72bc7-52f6-45a6-8118-eefba66e6992","Type":"ContainerStarted","Data":"46d1f327fadca73fefbade979cbed1898961fc8a448adfa638c6d1d3099e17a8"} Nov 03 06:59:21.449808 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.449782 4925 patch_prober.go:29] interesting pod/image-registry-54fc5dcf5f-l5q5j container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 06:59:21.449919 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.449821 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" podUID="f55630a1-1afe-446c-bda0-20bfba394fb5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:59:21.871497 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.871467 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57998bb766-8gfrg"] Nov 03 06:59:21.877305 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.877274 4925 patch_prober.go:29] interesting pod/image-registry-57998bb766-8gfrg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 06:59:21.877445 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:21.877328 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" podUID="063967c5-741c-46f1-a023-9384b59ccaa7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:59:22.055363 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:22.055331 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:22.271874 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:22.271792 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:22.478793 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:22.478759 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:22.531889 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:22.531817 4925 patch_prober.go:29] interesting pod/image-registry-54fc5dcf5f-l5q5j container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 06:59:22.531889 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:22.531862 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" podUID="f55630a1-1afe-446c-bda0-20bfba394fb5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 06:59:22.741652 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:22.741616 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:22.949994 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:22.949962 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:25.691727 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:25.691692 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:25.949393 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:25.949311 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:26.149315 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:26.149282 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:26.343502 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:26.343470 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:26.616719 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:26.616638 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dfpc7" Nov 03 06:59:30.518600 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:30.518567 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:30.729440 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:30.729410 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:31.215515 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:31.215482 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:31.379457 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:31.379433 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cpnw7" Nov 03 06:59:31.512516 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:31.512448 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:31.876081 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:31.876054 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:32.074749 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.074719 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:32.317491 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.317460 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:32.532159 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.532134 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 06:59:32.567180 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.567148 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:32.760755 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.760679 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7gqd4"] Nov 03 06:59:32.783197 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.783158 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.805978 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.805947 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:32.838963 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.838942 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 06:59:32.839151 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.839135 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 06:59:32.839198 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.839135 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-666h2\"" Nov 03 06:59:32.918221 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.918197 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 06:59:32.918557 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.918247 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 06:59:32.926548 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926530 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c81080-acdb-4468-8ef4-a2e35d7b2969-metrics-client-ca\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.926607 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926568 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.926607 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926594 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-wtmp\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.926738 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926721 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-root\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.926800 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926747 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-textfile\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.926800 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926763 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gmr\" (UniqueName: \"kubernetes.io/projected/99c81080-acdb-4468-8ef4-a2e35d7b2969-kube-api-access-m2gmr\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.926899 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926867 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.926954 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926922 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-tls\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:32.927003 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:32.926951 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-sys\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.027970 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.027913 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-root\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.027970 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.027939 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-textfile\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.027970 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.027955 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gmr\" (UniqueName: \"kubernetes.io/projected/99c81080-acdb-4468-8ef4-a2e35d7b2969-kube-api-access-m2gmr\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.027975 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028051 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-tls\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028083 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-sys\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028109 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c81080-acdb-4468-8ef4-a2e35d7b2969-metrics-client-ca\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028055 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-root\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028151 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028185 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-sys\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028224 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028194 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-wtmp\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028610 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028304 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-wtmp\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028610 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028473 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-textfile\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028695 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028622 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-accelerators-collector-config\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.028784 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.028766 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c81080-acdb-4468-8ef4-a2e35d7b2969-metrics-client-ca\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.030732 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.030709 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.030831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.030789 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99c81080-acdb-4468-8ef4-a2e35d7b2969-node-exporter-tls\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.092164 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.092142 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gmr\" (UniqueName: \"kubernetes.io/projected/99c81080-acdb-4468-8ef4-a2e35d7b2969-kube-api-access-m2gmr\") pod \"node-exporter-7gqd4\" (UID: \"99c81080-acdb-4468-8ef4-a2e35d7b2969\") " pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.133452 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.133426 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7gqd4" Nov 03 06:59:33.174792 ip-10-0-136-184 kubenswrapper[4925]: W1103 06:59:33.174771 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c81080_acdb_4468_8ef4_a2e35d7b2969.slice/crio-880e69ae6890bf45be6eff07e84a45dbea34ce4386aa1ae2a326e83df93403ed WatchSource:0}: Error finding container 880e69ae6890bf45be6eff07e84a45dbea34ce4386aa1ae2a326e83df93403ed: Status 404 returned error can't find the container with id 880e69ae6890bf45be6eff07e84a45dbea34ce4386aa1ae2a326e83df93403ed Nov 03 06:59:33.209361 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.209330 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:33.417048 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.416994 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gqd4" event={"ID":"99c81080-acdb-4468-8ef4-a2e35d7b2969","Type":"ContainerStarted","Data":"880e69ae6890bf45be6eff07e84a45dbea34ce4386aa1ae2a326e83df93403ed"} Nov 03 06:59:33.428752 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.428722 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:33.649180 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:33.649147 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:34.236348 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:34.236290 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:34.751354 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:34.751322 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:35.145362 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:35.145325 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:35.430074 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:35.429974 4925 generic.go:356] "Generic (PLEG): container finished" podID="99c81080-acdb-4468-8ef4-a2e35d7b2969" containerID="88a4107ae605a57ddb57eb74ff3aa95f7584fe5c5ca20d0b8d9178ef9a4f058b" exitCode=0 Nov 03 06:59:35.430477 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:35.430078 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gqd4" event={"ID":"99c81080-acdb-4468-8ef4-a2e35d7b2969","Type":"ContainerDied","Data":"88a4107ae605a57ddb57eb74ff3aa95f7584fe5c5ca20d0b8d9178ef9a4f058b"} Nov 03 06:59:36.437025 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:36.436972 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gqd4" event={"ID":"99c81080-acdb-4468-8ef4-a2e35d7b2969","Type":"ContainerStarted","Data":"492e6a8c92ae7b29e6f5687da5088f58327f79aa8d651f5f61cf06af929f30f1"} Nov 03 06:59:36.437413 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:36.437039 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7gqd4" event={"ID":"99c81080-acdb-4468-8ef4-a2e35d7b2969","Type":"ContainerStarted","Data":"05e1ac84bd3baff128e62fe6595bb5dc1b1cd8e43f33200c476c429a244e12b9"} Nov 03 06:59:36.563827 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:36.563779 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7gqd4" podStartSLOduration=2.58219227 podStartE2EDuration="4.563765616s" totalImagesPullingTime="1.895063398s" totalInitContainerRuntime="86.509948ms" isStatefulPod=true podCreationTimestamp="2026-11-03 06:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:36.563260298 +0000 UTC m=+36.496788955" watchObservedRunningTime="2026-11-03 06:59:36.563765616 +0000 UTC m=+36.497294272" Nov 03 06:59:37.399401 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:37.399361 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:37.705575 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:37.705497 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:37.981606 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:37.981527 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:39.137415 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:39.137369 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 06:59:41.420069 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:41.419984 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:41.699565 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:41.699476 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:41.949848 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:41.949763 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:42.217613 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:42.217537 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:42.557179 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:42.557094 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:43.598900 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:43.598866 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:43.845532 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:43.845498 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:44.091004 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:44.090974 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54fc5dcf5f-l5q5j"] Nov 03 06:59:44.314686 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:44.314657 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:44.629836 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:44.629804 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:44.898592 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:44.898513 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:45.545535 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:45.545500 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:45.775775 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:45.775743 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:46.895852 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:46.895807 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" podUID="063967c5-741c-46f1-a023-9384b59ccaa7" containerName="registry" containerID="cri-o://d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0" gracePeriod=30 Nov 03 06:59:47.038365 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.038333 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:47.124184 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.124161 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:47.240228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240166 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240196 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240228 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240216 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240478 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240243 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240478 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240312 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240478 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240353 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240518 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240638 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240561 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\") pod \"063967c5-741c-46f1-a023-9384b59ccaa7\" (UID: \"063967c5-741c-46f1-a023-9384b59ccaa7\") " Nov 03 06:59:47.240759 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240687 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:59:47.240820 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240755 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 06:59:47.240888 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240867 4925 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-registry-certificates\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.241128 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.240892 4925 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/063967c5-741c-46f1-a023-9384b59ccaa7-trusted-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.242847 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.242816 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:59:47.242963 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.242938 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 06:59:47.243178 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.243157 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:59:47.243444 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.243423 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "kube-api-access-gk9wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:59:47.243444 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.243422 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 06:59:47.247903 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.247882 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted" pod "063967c5-741c-46f1-a023-9384b59ccaa7" (UID: "063967c5-741c-46f1-a023-9384b59ccaa7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 06:59:47.342272 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.342242 4925 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/063967c5-741c-46f1-a023-9384b59ccaa7-ca-trust-extracted\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.342272 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.342273 4925 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-bound-sa-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.342456 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.342287 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk9wc\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-kube-api-access-gk9wc\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.342456 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.342302 4925 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-installation-pull-secrets\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.342456 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.342316 4925 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/063967c5-741c-46f1-a023-9384b59ccaa7-image-registry-private-configuration\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.342456 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.342329 4925 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/063967c5-741c-46f1-a023-9384b59ccaa7-registry-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 06:59:47.444402 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.444366 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:47.479679 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.479652 4925 generic.go:356] "Generic (PLEG): container finished" podID="063967c5-741c-46f1-a023-9384b59ccaa7" containerID="d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0" exitCode=0 Nov 03 06:59:47.479813 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.479684 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" event={"ID":"063967c5-741c-46f1-a023-9384b59ccaa7","Type":"ContainerDied","Data":"d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0"} Nov 03 06:59:47.479813 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.479705 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" Nov 03 06:59:47.479813 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.479718 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57998bb766-8gfrg" event={"ID":"063967c5-741c-46f1-a023-9384b59ccaa7","Type":"ContainerDied","Data":"27d2a46b9ce4f48d9fadf25171125e8da044a157536ffaba7292db0a0b58ef3f"} Nov 03 06:59:47.479813 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.479738 4925 scope.go:122] "RemoveContainer" containerID="d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0" Nov 03 06:59:47.489203 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.489187 4925 scope.go:122] "RemoveContainer" containerID="d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0" Nov 03 06:59:47.489449 ip-10-0-136-184 kubenswrapper[4925]: E1103 06:59:47.489430 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0\": container with ID starting with d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0 not found: ID does not exist" containerID="d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0" Nov 03 06:59:47.489504 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.489457 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0"} err="failed to get container status \"d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0\": rpc error: code = NotFound desc = could not find container \"d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0\": container with ID starting with d517636c5885e1989d640ec2a1190cb367f1876e4a88d3c3f4473762f280a0a0 not found: ID does not exist" Nov 03 06:59:47.651515 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.651492 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57998bb766-8gfrg"] Nov 03 06:59:47.658586 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:47.658562 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57998bb766-8gfrg"] Nov 03 06:59:48.188831 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:48.188801 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063967c5-741c-46f1-a023-9384b59ccaa7" path="/var/lib/kubelet/pods/063967c5-741c-46f1-a023-9384b59ccaa7/volumes" Nov 03 06:59:48.429349 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:48.429313 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:48.655483 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:48.655452 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:50.226927 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:50.226895 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 06:59:50.494454 ip-10-0-136-184 kubenswrapper[4925]: I1103 06:59:50.494361 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:00.509262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:00.509175 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:00.730243 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:00.730208 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:03.388053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:03.387991 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:03.646466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:03.646383 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:09.112200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.112132 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" podUID="f55630a1-1afe-446c-bda0-20bfba394fb5" containerName="registry" containerID="cri-o://fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b" gracePeriod=30 Nov 03 07:00:09.138174 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:00:09.138141 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:00:09.356164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.356141 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 07:00:09.404308 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404248 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404308 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404284 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404312 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404330 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404353 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404386 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404423 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404459 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\") pod \"f55630a1-1afe-446c-bda0-20bfba394fb5\" (UID: \"f55630a1-1afe-446c-bda0-20bfba394fb5\") " Nov 03 07:00:09.404893 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.404857 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:09.405122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.405041 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:09.406859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.406761 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:09.406984 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.406935 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "kube-api-access-rrjh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:00:09.407081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.407062 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:00:09.407402 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.407371 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:09.407402 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.407384 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:00:09.411854 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.411830 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted" pod "f55630a1-1afe-446c-bda0-20bfba394fb5" (UID: "f55630a1-1afe-446c-bda0-20bfba394fb5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:00:09.505803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505779 4925 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-image-registry-private-configuration\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.505803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505804 4925 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f55630a1-1afe-446c-bda0-20bfba394fb5-ca-trust-extracted\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.505931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505815 4925 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-trusted-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.505931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505823 4925 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.505931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505832 4925 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f55630a1-1afe-446c-bda0-20bfba394fb5-installation-pull-secrets\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.505931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505841 4925 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f55630a1-1afe-446c-bda0-20bfba394fb5-registry-certificates\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.505931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505850 4925 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-bound-sa-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.505931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.505858 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrjh6\" (UniqueName: \"kubernetes.io/projected/f55630a1-1afe-446c-bda0-20bfba394fb5-kube-api-access-rrjh6\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:09.563050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.562996 4925 generic.go:356] "Generic (PLEG): container finished" podID="f55630a1-1afe-446c-bda0-20bfba394fb5" containerID="fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b" exitCode=0 Nov 03 07:00:09.563156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.563060 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" Nov 03 07:00:09.563156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.563078 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" event={"ID":"f55630a1-1afe-446c-bda0-20bfba394fb5","Type":"ContainerDied","Data":"fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b"} Nov 03 07:00:09.563156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.563123 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54fc5dcf5f-l5q5j" event={"ID":"f55630a1-1afe-446c-bda0-20bfba394fb5","Type":"ContainerDied","Data":"7a9a38e9265dc7076087c01009c66c6261abbb12576fc1e59bc205ca4b9e0653"} Nov 03 07:00:09.563156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.563146 4925 scope.go:122] "RemoveContainer" containerID="fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b" Nov 03 07:00:09.571522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.571499 4925 scope.go:122] "RemoveContainer" containerID="fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b" Nov 03 07:00:09.571796 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:00:09.571775 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b\": container with ID starting with fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b not found: ID does not exist" containerID="fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b" Nov 03 07:00:09.571881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.571803 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b"} err="failed to get container status \"fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b\": rpc error: code = NotFound desc = could not find container \"fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b\": container with ID starting with fe3e151c4c7c64ecef9b39818b387632176009348de492f2431f494de18f4f7b not found: ID does not exist" Nov 03 07:00:09.734768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.734693 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54fc5dcf5f-l5q5j"] Nov 03 07:00:09.740983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:09.740961 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54fc5dcf5f-l5q5j"] Nov 03 07:00:10.187763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:10.187729 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55630a1-1afe-446c-bda0-20bfba394fb5" path="/var/lib/kubelet/pods/f55630a1-1afe-446c-bda0-20bfba394fb5/volumes" Nov 03 07:00:11.573203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:11.573168 4925 generic.go:356] "Generic (PLEG): container finished" podID="728637b6-bbb0-4532-adde-4555c4f4153d" containerID="f08c6194fe153a992f77cf467b10d686a1f673c5b4c4f091e149bf9022f09f54" exitCode=0 Nov 03 07:00:11.573585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:11.573235 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-299kl" event={"ID":"728637b6-bbb0-4532-adde-4555c4f4153d","Type":"ContainerDied","Data":"f08c6194fe153a992f77cf467b10d686a1f673c5b4c4f091e149bf9022f09f54"} Nov 03 07:00:11.573628 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:11.573583 4925 scope.go:122] "RemoveContainer" containerID="f08c6194fe153a992f77cf467b10d686a1f673c5b4c4f091e149bf9022f09f54" Nov 03 07:00:12.414825 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:12.414789 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:12.578891 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:12.578858 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-299kl" event={"ID":"728637b6-bbb0-4532-adde-4555c4f4153d","Type":"ContainerStarted","Data":"5e11a0afd144c7bbc641ea3d26a8c852631976b9633234e5006c9449018d052f"} Nov 03 07:00:12.626908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:12.626873 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:15.499461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:15.499424 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:15.750145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:15.750055 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:16.594488 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:16.594459 4925 generic.go:356] "Generic (PLEG): container finished" podID="362105ce-9f40-4ad3-964b-472512bf9518" containerID="ef67d3a9f2e404168899a7121231c2f013d90ac7dd44c7bd3e11218dc598e2e9" exitCode=0 Nov 03 07:00:16.594860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:16.594522 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" event={"ID":"362105ce-9f40-4ad3-964b-472512bf9518","Type":"ContainerDied","Data":"ef67d3a9f2e404168899a7121231c2f013d90ac7dd44c7bd3e11218dc598e2e9"} Nov 03 07:00:16.594860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:16.594805 4925 scope.go:122] "RemoveContainer" containerID="ef67d3a9f2e404168899a7121231c2f013d90ac7dd44c7bd3e11218dc598e2e9" Nov 03 07:00:17.599656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:17.599624 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6dvwz" event={"ID":"362105ce-9f40-4ad3-964b-472512bf9518","Type":"ContainerStarted","Data":"157f62fe5cac79a3e4dde43914359525090b9ed7500bbe5abce773e75c3af284"} Nov 03 07:00:31.651383 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:31.651348 4925 generic.go:356] "Generic (PLEG): container finished" podID="271984b8-d201-41a3-a545-690f8a9fad89" containerID="a359b80fe77b0170ae2238d20d89b0862f04ba267047e2e813e15c3a8c17b22e" exitCode=0 Nov 03 07:00:31.651927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:31.651433 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" event={"ID":"271984b8-d201-41a3-a545-690f8a9fad89","Type":"ContainerDied","Data":"a359b80fe77b0170ae2238d20d89b0862f04ba267047e2e813e15c3a8c17b22e"} Nov 03 07:00:31.651927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:31.651837 4925 scope.go:122] "RemoveContainer" containerID="a359b80fe77b0170ae2238d20d89b0862f04ba267047e2e813e15c3a8c17b22e" Nov 03 07:00:32.656651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:32.656618 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4n4t6" event={"ID":"271984b8-d201-41a3-a545-690f8a9fad89","Type":"ContainerStarted","Data":"c3f970f6c5b6e0f62ec035f2475a0f8de4cb654e02c1408e5dcc92cae9239cb3"} Nov 03 07:00:37.138506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:37.138473 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:37.408886 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:37.408771 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:39.136645 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:00:39.136605 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:00:47.361910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:47.361873 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:47.612904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:47.612816 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:47.816276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:47.816238 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:00:50.546842 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:00:50.546796 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:01:09.136688 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:01:09.136645 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:01:39.137447 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:01:39.137405 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:01:50.915769 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:01:50.915730 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:01:55.019454 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:01:55.019412 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:07.693126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:07.693087 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:07.905721 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:07.905678 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:09.137681 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:02:09.137635 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:02:15.430718 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:15.430679 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:15.697333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:15.697247 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:16.688135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:16.688104 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:16.897639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:16.897601 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:39.136419 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:02:39.136385 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:02:51.260169 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:51.260089 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:02:54.895130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:02:54.895088 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:03:09.137525 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:03:09.137490 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:03:28.319886 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:03:28.319853 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:03:31.567954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:03:31.567922 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:03:31.817119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:03:31.817077 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:03:39.136158 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:03:39.136122 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:03:51.610430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:03:51.610389 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:00.190931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:00.190895 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:04:00.191637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:00.191617 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:04:00.195346 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:00.195323 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:04:00.195885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:00.195865 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:04:09.136054 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:04:09.135996 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:04:25.606104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:25.605993 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:28.509888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:28.509849 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:39.136686 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:04:39.136646 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:04:45.554257 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:45.554212 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:45.765576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:45.765539 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:47.216554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:47.216502 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:47.468197 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:47.468094 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:48.621528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:48.621488 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:50.727891 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.727861 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t"] Nov 03 07:04:50.728279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.728193 4925 cpu_manager.go:414] "containerMap: removing container" podUID="063967c5-741c-46f1-a023-9384b59ccaa7" containerName="registry" Nov 03 07:04:50.728279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.728205 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063967c5-741c-46f1-a023-9384b59ccaa7" containerName="registry" Nov 03 07:04:50.728279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.728215 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f55630a1-1afe-446c-bda0-20bfba394fb5" containerName="registry" Nov 03 07:04:50.728279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.728221 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55630a1-1afe-446c-bda0-20bfba394fb5" containerName="registry" Nov 03 07:04:50.728279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.728277 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="063967c5-741c-46f1-a023-9384b59ccaa7" containerName="registry" Nov 03 07:04:50.728435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.728285 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55630a1-1afe-446c-bda0-20bfba394fb5" containerName="registry" Nov 03 07:04:50.731271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.731253 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.755683 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.755660 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.755786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.755691 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2tg\" (UniqueName: \"kubernetes.io/projected/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-kube-api-access-5j2tg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.755786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.755716 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.836612 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.836587 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7blss\"" Nov 03 07:04:50.836612 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.836585 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 07:04:50.836784 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.836585 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 07:04:50.837601 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.837583 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t"] Nov 03 07:04:50.856881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.856853 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2tg\" (UniqueName: \"kubernetes.io/projected/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-kube-api-access-5j2tg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.857003 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.856889 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.857003 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.856939 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.857291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.857276 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.857329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.857312 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:50.921859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:50.921826 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2tg\" (UniqueName: \"kubernetes.io/projected/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-kube-api-access-5j2tg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:51.045869 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.045811 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:04:51.419703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.419666 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t"] Nov 03 07:04:51.425830 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:04:51.425806 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9ce6c0_02e1_4c08_85d3_6a45ed63df6e.slice/crio-a4e923e234371576d895d4b7f05e5eb26a1160e5970f695d8a9f57289e9c9cd7 WatchSource:0}: Error finding container a4e923e234371576d895d4b7f05e5eb26a1160e5970f695d8a9f57289e9c9cd7: Status 404 returned error can't find the container with id a4e923e234371576d895d4b7f05e5eb26a1160e5970f695d8a9f57289e9c9cd7 Nov 03 07:04:51.427798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.427779 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:04:51.460255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.460229 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx"] Nov 03 07:04:51.464517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.464502 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.522999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.522966 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-2h2l4\"" Nov 03 07:04:51.523396 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.523369 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx"] Nov 03 07:04:51.561252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.561225 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" event={"ID":"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e","Type":"ContainerStarted","Data":"a4e923e234371576d895d4b7f05e5eb26a1160e5970f695d8a9f57289e9c9cd7"} Nov 03 07:04:51.563655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.563638 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdlp\" (UniqueName: \"kubernetes.io/projected/9232d701-19e3-47df-9631-9af0a66b6572-kube-api-access-xhdlp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.563713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.563668 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.563713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.563686 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.568053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.568031 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw"] Nov 03 07:04:51.571255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.571241 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.625271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.625255 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dck52\"" Nov 03 07:04:51.631355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.631327 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw"] Nov 03 07:04:51.660210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.660185 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7"] Nov 03 07:04:51.663700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.663643 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.664208 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664184 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.664306 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664238 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdlp\" (UniqueName: \"kubernetes.io/projected/9232d701-19e3-47df-9631-9af0a66b6572-kube-api-access-xhdlp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.664306 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664262 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.664306 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664277 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.664306 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664297 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkm9\" (UniqueName: \"kubernetes.io/projected/f563e399-e023-42fe-8fc7-4bb2a3d89153-kube-api-access-rpkm9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.664528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664313 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.664851 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664829 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.664950 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.664891 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.719225 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.719204 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-wrr7c\"" Nov 03 07:04:51.724642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.724619 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7"] Nov 03 07:04:51.730414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.730394 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdlp\" (UniqueName: \"kubernetes.io/projected/9232d701-19e3-47df-9631-9af0a66b6572-kube-api-access-xhdlp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.759839 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.759818 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46"] Nov 03 07:04:51.763227 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.763213 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.765441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765420 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrgtr\" (UniqueName: \"kubernetes.io/projected/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-kube-api-access-zrgtr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.765528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765453 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkm9\" (UniqueName: \"kubernetes.io/projected/f563e399-e023-42fe-8fc7-4bb2a3d89153-kube-api-access-rpkm9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.765528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765471 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.765645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765561 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.765645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765621 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.765751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765686 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.765879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765860 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.765950 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.765933 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.779418 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.779400 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:04:51.819934 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.819901 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-stz4q\"" Nov 03 07:04:51.824837 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.824734 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46"] Nov 03 07:04:51.836610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.836212 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkm9\" (UniqueName: \"kubernetes.io/projected/f563e399-e023-42fe-8fc7-4bb2a3d89153-kube-api-access-rpkm9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.866684 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.866656 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.866820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.866692 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgtr\" (UniqueName: \"kubernetes.io/projected/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-kube-api-access-zrgtr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.866820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.866716 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.866820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.866732 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.866939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.866840 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6rr\" (UniqueName: \"kubernetes.io/projected/18dc22e4-c6e6-4055-a715-c6a59340e140-kube-api-access-kq6rr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.866939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.866882 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.867127 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.867108 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.867191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.867177 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.886670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.886643 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:04:51.934315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.934240 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrgtr\" (UniqueName: \"kubernetes.io/projected/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-kube-api-access-zrgtr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:51.965056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.964998 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:51.967888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.967861 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.967888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.967889 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.968097 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.967918 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6rr\" (UniqueName: \"kubernetes.io/projected/18dc22e4-c6e6-4055-a715-c6a59340e140-kube-api-access-kq6rr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.968252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.968234 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.968326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.968257 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:51.979838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:51.979817 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:04:52.036677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.035658 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6rr\" (UniqueName: \"kubernetes.io/projected/18dc22e4-c6e6-4055-a715-c6a59340e140-kube-api-access-kq6rr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:52.079085 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.079057 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:04:52.162647 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.162621 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx"] Nov 03 07:04:52.164939 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:04:52.164912 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9232d701_19e3_47df_9631_9af0a66b6572.slice/crio-959047cd39a86fe0609d74346667d96c33189d2ca1ee63f437fd1e38e9e5560f WatchSource:0}: Error finding container 959047cd39a86fe0609d74346667d96c33189d2ca1ee63f437fd1e38e9e5560f: Status 404 returned error can't find the container with id 959047cd39a86fe0609d74346667d96c33189d2ca1ee63f437fd1e38e9e5560f Nov 03 07:04:52.264558 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.264535 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw"] Nov 03 07:04:52.266344 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:04:52.266323 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf563e399_e023_42fe_8fc7_4bb2a3d89153.slice/crio-b189612d7931f82f90ee67b35c58574bde8b3b31cb3c63d9c140e6861f0a025b WatchSource:0}: Error finding container b189612d7931f82f90ee67b35c58574bde8b3b31cb3c63d9c140e6861f0a025b: Status 404 returned error can't find the container with id b189612d7931f82f90ee67b35c58574bde8b3b31cb3c63d9c140e6861f0a025b Nov 03 07:04:52.351598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.351577 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7"] Nov 03 07:04:52.353568 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:04:52.353545 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a2ed70_0fc9_4a8e_8971_49e50a885fd2.slice/crio-cbe5c63fef988bdf80ab72675933b47ecb50dc16504ab4e7d89ae1c5acb9cfc9 WatchSource:0}: Error finding container cbe5c63fef988bdf80ab72675933b47ecb50dc16504ab4e7d89ae1c5acb9cfc9: Status 404 returned error can't find the container with id cbe5c63fef988bdf80ab72675933b47ecb50dc16504ab4e7d89ae1c5acb9cfc9 Nov 03 07:04:52.453824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.453803 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46"] Nov 03 07:04:52.455816 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:04:52.455788 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dc22e4_c6e6_4055_a715_c6a59340e140.slice/crio-22d2a951b0670cd5adf6e9d6eff34f7e5aba627e389957b313adf84cc0aa6f6c WatchSource:0}: Error finding container 22d2a951b0670cd5adf6e9d6eff34f7e5aba627e389957b313adf84cc0aa6f6c: Status 404 returned error can't find the container with id 22d2a951b0670cd5adf6e9d6eff34f7e5aba627e389957b313adf84cc0aa6f6c Nov 03 07:04:52.566786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.566750 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" event={"ID":"63a2ed70-0fc9-4a8e-8971-49e50a885fd2","Type":"ContainerStarted","Data":"cbe5c63fef988bdf80ab72675933b47ecb50dc16504ab4e7d89ae1c5acb9cfc9"} Nov 03 07:04:52.568248 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.568191 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" event={"ID":"f563e399-e023-42fe-8fc7-4bb2a3d89153","Type":"ContainerStarted","Data":"b189612d7931f82f90ee67b35c58574bde8b3b31cb3c63d9c140e6861f0a025b"} Nov 03 07:04:52.569478 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.569455 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" event={"ID":"9232d701-19e3-47df-9631-9af0a66b6572","Type":"ContainerStarted","Data":"959047cd39a86fe0609d74346667d96c33189d2ca1ee63f437fd1e38e9e5560f"} Nov 03 07:04:52.570801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:52.570780 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" event={"ID":"18dc22e4-c6e6-4055-a715-c6a59340e140","Type":"ContainerStarted","Data":"22d2a951b0670cd5adf6e9d6eff34f7e5aba627e389957b313adf84cc0aa6f6c"} Nov 03 07:04:53.312358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.312324 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh"] Nov 03 07:04:53.316942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.316356 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.378898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.378837 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.378898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.378881 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.379134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.378952 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5g2\" (UniqueName: \"kubernetes.io/projected/43851410-de5e-4d41-8ff3-f2601a30dd86-kube-api-access-4n5g2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.423727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.423695 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-qg5wx\"" Nov 03 07:04:53.425586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.425521 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh"] Nov 03 07:04:53.480700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.480649 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5g2\" (UniqueName: \"kubernetes.io/projected/43851410-de5e-4d41-8ff3-f2601a30dd86-kube-api-access-4n5g2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.480878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.480748 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.480982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.480966 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.485895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.485801 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.489112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.487447 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.556991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.556946 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5g2\" (UniqueName: \"kubernetes.io/projected/43851410-de5e-4d41-8ff3-f2601a30dd86-kube-api-access-4n5g2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.637608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.637574 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:04:53.881232 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:53.881197 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:54.062947 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.062881 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh"] Nov 03 07:04:54.175231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.175122 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:54.371484 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.371409 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6"] Nov 03 07:04:54.376315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.376290 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.388866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.388829 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.388866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.388879 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.389168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.388929 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zncl\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-kube-api-access-7zncl\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.389168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.388973 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.389168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.388998 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.389168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.389052 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e5265-25ca-4578-9533-096c5818509d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.389168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.389084 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.389168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.389113 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e5265-25ca-4578-9533-096c5818509d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.389168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.389162 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.436261 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.436234 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 07:04:54.438289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.438247 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89ggc\"" Nov 03 07:04:54.492846 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.492815 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6"] Nov 03 07:04:54.493259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493238 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493279 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493324 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zncl\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-kube-api-access-7zncl\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493365 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493607 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493388 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493607 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493427 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e5265-25ca-4578-9533-096c5818509d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493607 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493458 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493607 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493488 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e5265-25ca-4578-9533-096c5818509d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493607 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493531 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.493995 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.493903 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.494164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.494142 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.494837 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.494811 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.495136 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.495084 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.496171 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.496150 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e5265-25ca-4578-9533-096c5818509d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.502219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.502194 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.514158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.514066 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e5265-25ca-4578-9533-096c5818509d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.528800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.528763 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:54.565213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.565182 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.567909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.567870 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zncl\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-kube-api-access-7zncl\") pod \"maas-default-gateway-openshift-default-58d898bd55-zlfl6\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.590499 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.590421 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" event={"ID":"43851410-de5e-4d41-8ff3-f2601a30dd86","Type":"ContainerStarted","Data":"c4f29a9cad3d5248b818a0657096c6cae8550a06609fcf65d445f7d139e07736"} Nov 03 07:04:54.696238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.696146 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:04:54.723169 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.720510 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g"] Nov 03 07:04:54.729295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.727131 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp"] Nov 03 07:04:54.729295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.728284 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.740248 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.740227 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.787661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.785117 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g"] Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797348 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797389 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797409 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797432 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xrj\" (UniqueName: \"kubernetes.io/projected/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-kube-api-access-44xrj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797462 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797484 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797522 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klr4w\" (UniqueName: \"kubernetes.io/projected/a0933a67-48ab-4a25-9e4b-c02b650eefcf-kube-api-access-klr4w\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797546 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797567 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797595 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797623 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.797829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.797655 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.836457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.836279 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-dp5nm\"" Nov 03 07:04:54.836457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.836424 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z2v9m\"" Nov 03 07:04:54.872032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.871841 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp"] Nov 03 07:04:54.875278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.875094 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898356 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898418 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898472 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898520 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898557 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898616 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898647 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xrj\" (UniqueName: \"kubernetes.io/projected/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-kube-api-access-44xrj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898686 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898761 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898802 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klr4w\" (UniqueName: \"kubernetes.io/projected/a0933a67-48ab-4a25-9e4b-c02b650eefcf-kube-api-access-klr4w\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898880 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.898915 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.899492 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.899764 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.900142 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.900449 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.900732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.900717 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.901844 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.901260 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.903586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.902591 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.903994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.903963 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.904543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.904518 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.967916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.967825 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xrj\" (UniqueName: \"kubernetes.io/projected/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-kube-api-access-44xrj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:54.968065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.967914 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klr4w\" (UniqueName: \"kubernetes.io/projected/a0933a67-48ab-4a25-9e4b-c02b650eefcf-kube-api-access-klr4w\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:54.969385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:54.969358 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0933a67-48ab-4a25-9e4b-c02b650eefcf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp\" (UID: \"a0933a67-48ab-4a25-9e4b-c02b650eefcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:55.060562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.060300 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:04:55.072377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.072352 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:04:55.128868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.128756 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:04:55.189655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.189527 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6"] Nov 03 07:04:55.514421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.514325 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g"] Nov 03 07:04:55.521837 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.520983 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp"] Nov 03 07:04:55.526629 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:04:55.526592 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6030cf_b2ed_46d7_9c7e_c75d386fe4a9.slice/crio-d8691ee95935b609063d4abfae76636493502b963e368877a5edfa3d4b7625ff WatchSource:0}: Error finding container d8691ee95935b609063d4abfae76636493502b963e368877a5edfa3d4b7625ff: Status 404 returned error can't find the container with id d8691ee95935b609063d4abfae76636493502b963e368877a5edfa3d4b7625ff Nov 03 07:04:55.528093 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:04:55.528059 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0933a67_48ab_4a25_9e4b_c02b650eefcf.slice/crio-72c2bd77885d5eda5c5b47313cf2e99f065ff79b16b60b4e63bbe739038edcce WatchSource:0}: Error finding container 72c2bd77885d5eda5c5b47313cf2e99f065ff79b16b60b4e63bbe739038edcce: Status 404 returned error can't find the container with id 72c2bd77885d5eda5c5b47313cf2e99f065ff79b16b60b4e63bbe739038edcce Nov 03 07:04:55.602048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.601933 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" event={"ID":"a0933a67-48ab-4a25-9e4b-c02b650eefcf","Type":"ContainerStarted","Data":"72c2bd77885d5eda5c5b47313cf2e99f065ff79b16b60b4e63bbe739038edcce"} Nov 03 07:04:55.605446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.605413 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" event={"ID":"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9","Type":"ContainerStarted","Data":"d8691ee95935b609063d4abfae76636493502b963e368877a5edfa3d4b7625ff"} Nov 03 07:04:55.609988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:55.609926 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" event={"ID":"da9e5265-25ca-4578-9533-096c5818509d","Type":"ContainerStarted","Data":"a395dae042e5e0b366545fc649f426f672fddc0de849f30fe9334e471730e771"} Nov 03 07:04:58.641775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:04:58.641738 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:05.189124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:05.189077 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:05:05.189520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:05.189206 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:05:05.189520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:05.189263 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:05:05.700780 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:05.700059 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" event={"ID":"a0933a67-48ab-4a25-9e4b-c02b650eefcf","Type":"ContainerStarted","Data":"2d5769e798f4d181c0f7a012c93858f059515ca1c1fdf0508f36ff783e5a8423"} Nov 03 07:05:05.881184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:05.880925 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" podStartSLOduration=2.226118767 podStartE2EDuration="11.880903787s" totalImagesPullingTime="9.65478502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:05.879075328 +0000 UTC m=+365.812603986" watchObservedRunningTime="2026-11-03 07:05:05.880903787 +0000 UTC m=+365.814432445" Nov 03 07:05:06.073059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.072764 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:05:06.079706 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.079449 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:05:06.709154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.709105 4925 generic.go:356] "Generic (PLEG): container finished" podID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerID="99bf22ce08706330a89d9897121874e3477e792325e7cc514abbd1994746a919" exitCode=0 Nov 03 07:05:06.710190 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.710155 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" event={"ID":"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9","Type":"ContainerDied","Data":"99bf22ce08706330a89d9897121874e3477e792325e7cc514abbd1994746a919"} Nov 03 07:05:06.715695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.714727 4925 generic.go:356] "Generic (PLEG): container finished" podID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerID="be42f6021436ff3d43aea1eb35dec96c3777b32934b77e1603ab3481dcaa30be" exitCode=0 Nov 03 07:05:06.716515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.716062 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" event={"ID":"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e","Type":"ContainerDied","Data":"be42f6021436ff3d43aea1eb35dec96c3777b32934b77e1603ab3481dcaa30be"} Nov 03 07:05:06.716515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.716109 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:05:06.717578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:06.717085 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp" Nov 03 07:05:08.725828 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.725750 4925 generic.go:356] "Generic (PLEG): container finished" podID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerID="7f29082819e36785f032826da0968341df6ab2cb9b4e2cdfc2023a9ff5f8b806" exitCode=0 Nov 03 07:05:08.726273 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.725852 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" event={"ID":"63a2ed70-0fc9-4a8e-8971-49e50a885fd2","Type":"ContainerDied","Data":"7f29082819e36785f032826da0968341df6ab2cb9b4e2cdfc2023a9ff5f8b806"} Nov 03 07:05:08.727924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.727888 4925 generic.go:356] "Generic (PLEG): container finished" podID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerID="b01e995c1fd32857a8877b9575a699ea6cf536a7fe56ad6bf9bd0ecdbe0114f6" exitCode=0 Nov 03 07:05:08.727990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.727971 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" event={"ID":"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e","Type":"ContainerDied","Data":"b01e995c1fd32857a8877b9575a699ea6cf536a7fe56ad6bf9bd0ecdbe0114f6"} Nov 03 07:05:08.729652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.729630 4925 generic.go:356] "Generic (PLEG): container finished" podID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerID="3dd5655e8ef991821b91788e4a23670134d8bbe39e85fd0d24413ea09c7626e9" exitCode=0 Nov 03 07:05:08.729745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.729709 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" event={"ID":"f563e399-e023-42fe-8fc7-4bb2a3d89153","Type":"ContainerDied","Data":"3dd5655e8ef991821b91788e4a23670134d8bbe39e85fd0d24413ea09c7626e9"} Nov 03 07:05:08.732419 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.732395 4925 generic.go:356] "Generic (PLEG): container finished" podID="9232d701-19e3-47df-9631-9af0a66b6572" containerID="fa077d93a6c68cfd973735f2a9b8eca482e94e19daca73f6c12936b7c818800f" exitCode=0 Nov 03 07:05:08.732709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.732639 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" event={"ID":"9232d701-19e3-47df-9631-9af0a66b6572","Type":"ContainerDied","Data":"fa077d93a6c68cfd973735f2a9b8eca482e94e19daca73f6c12936b7c818800f"} Nov 03 07:05:08.734144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.734119 4925 generic.go:356] "Generic (PLEG): container finished" podID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerID="3d1a9cc804cb377c96641e7ecf9497e2d40df20354d9c9f19e0c9020472ced09" exitCode=0 Nov 03 07:05:08.734238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.734201 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" event={"ID":"43851410-de5e-4d41-8ff3-f2601a30dd86","Type":"ContainerDied","Data":"3d1a9cc804cb377c96641e7ecf9497e2d40df20354d9c9f19e0c9020472ced09"} Nov 03 07:05:08.736248 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.736225 4925 generic.go:356] "Generic (PLEG): container finished" podID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerID="6c31e82fa0727d9ac717acf979bb09eb95e26c15540a59b1a13283b2e0c5b041" exitCode=0 Nov 03 07:05:08.736408 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:08.736357 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" event={"ID":"18dc22e4-c6e6-4055-a715-c6a59340e140","Type":"ContainerDied","Data":"6c31e82fa0727d9ac717acf979bb09eb95e26c15540a59b1a13283b2e0c5b041"} Nov 03 07:05:09.050001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:09.049963 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:05:09.050139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:09.050060 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:05:09.050139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:09.050091 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:05:09.139422 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:05:09.139385 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:05:09.743229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:09.743152 4925 generic.go:356] "Generic (PLEG): container finished" podID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerID="a994303ca563a593421f838f1b0635ee533fb7bb5811b08cb78fead3654bb3f1" exitCode=0 Nov 03 07:05:09.743653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:09.743247 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" event={"ID":"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9","Type":"ContainerDied","Data":"a994303ca563a593421f838f1b0635ee533fb7bb5811b08cb78fead3654bb3f1"} Nov 03 07:05:09.744968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:09.744945 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" event={"ID":"da9e5265-25ca-4578-9533-096c5818509d","Type":"ContainerStarted","Data":"bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91"} Nov 03 07:05:09.991800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:09.991744 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podStartSLOduration=2.142006705 podStartE2EDuration="15.991728083s" totalImagesPullingTime="13.849721378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:09.990802536 +0000 UTC m=+369.924331192" watchObservedRunningTime="2026-11-03 07:05:09.991728083 +0000 UTC m=+369.925256808" Nov 03 07:05:10.696700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.696584 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:05:10.701958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.701893 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:05:10.756565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.755539 4925 generic.go:356] "Generic (PLEG): container finished" podID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerID="88e53417eb21f058f142d5f400c6fb20ad20b20f9ed06e32071f9bf1b6858330" exitCode=0 Nov 03 07:05:10.756565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.755657 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" event={"ID":"f563e399-e023-42fe-8fc7-4bb2a3d89153","Type":"ContainerDied","Data":"88e53417eb21f058f142d5f400c6fb20ad20b20f9ed06e32071f9bf1b6858330"} Nov 03 07:05:10.760246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.760210 4925 generic.go:356] "Generic (PLEG): container finished" podID="9232d701-19e3-47df-9631-9af0a66b6572" containerID="1189687cda63dceab23691da8870e5c893f72fd321856c225a426ea074133ac3" exitCode=0 Nov 03 07:05:10.760347 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.760304 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" event={"ID":"9232d701-19e3-47df-9631-9af0a66b6572","Type":"ContainerDied","Data":"1189687cda63dceab23691da8870e5c893f72fd321856c225a426ea074133ac3"} Nov 03 07:05:10.766293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.766262 4925 generic.go:356] "Generic (PLEG): container finished" podID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerID="ff9bf67293692487fc859c2e176944fa380cfb5fa36aaa25e01fbd49b8768ff1" exitCode=0 Nov 03 07:05:10.766404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.766383 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" event={"ID":"18dc22e4-c6e6-4055-a715-c6a59340e140","Type":"ContainerDied","Data":"ff9bf67293692487fc859c2e176944fa380cfb5fa36aaa25e01fbd49b8768ff1"} Nov 03 07:05:10.771515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.770690 4925 generic.go:356] "Generic (PLEG): container finished" podID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerID="38a1aaa98c68bbbfc977e9cc7cd4ee554c05c0fc51ef11776131145e9e71bf5f" exitCode=0 Nov 03 07:05:10.771515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.771493 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" event={"ID":"63a2ed70-0fc9-4a8e-8971-49e50a885fd2","Type":"ContainerDied","Data":"38a1aaa98c68bbbfc977e9cc7cd4ee554c05c0fc51ef11776131145e9e71bf5f"} Nov 03 07:05:10.771670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.771535 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:05:10.772460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:10.772443 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:05:12.806728 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:12.806698 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" event={"ID":"43851410-de5e-4d41-8ff3-f2601a30dd86","Type":"ContainerStarted","Data":"541f9feccffff128f2eb4bf1480d03c120698964ee2be7899ecea88006c34cdd"} Nov 03 07:05:13.816076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:13.816039 4925 generic.go:356] "Generic (PLEG): container finished" podID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerID="541f9feccffff128f2eb4bf1480d03c120698964ee2be7899ecea88006c34cdd" exitCode=0 Nov 03 07:05:13.816554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:13.816103 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" event={"ID":"43851410-de5e-4d41-8ff3-f2601a30dd86","Type":"ContainerDied","Data":"541f9feccffff128f2eb4bf1480d03c120698964ee2be7899ecea88006c34cdd"} Nov 03 07:05:23.917506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:23.916413 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" event={"ID":"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e","Type":"ContainerStarted","Data":"e536d331449763aea0da70e2f381956a09fff6a42d8952a873f03101717ed4fd"} Nov 03 07:05:24.041649 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:24.041580 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" podStartSLOduration=3.129934694 podStartE2EDuration="34.041560128s" totalImagesPullingTime="30.911625434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:24.040706635 +0000 UTC m=+383.974235314" watchObservedRunningTime="2026-11-03 07:05:24.041560128 +0000 UTC m=+383.975088786" Nov 03 07:05:24.935352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:24.934669 4925 generic.go:356] "Generic (PLEG): container finished" podID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerID="e536d331449763aea0da70e2f381956a09fff6a42d8952a873f03101717ed4fd" exitCode=0 Nov 03 07:05:24.935352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:24.934844 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" event={"ID":"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e","Type":"ContainerDied","Data":"e536d331449763aea0da70e2f381956a09fff6a42d8952a873f03101717ed4fd"} Nov 03 07:05:24.940750 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:24.940682 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" event={"ID":"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9","Type":"ContainerStarted","Data":"32b62f8175b2f7926e3869d64b2159cb3307e75ab857fafc4b3bb73bffe90017"} Nov 03 07:05:25.196888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:25.195147 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" podStartSLOduration=2.7718941089999998 podStartE2EDuration="31.195127544s" totalImagesPullingTime="28.423233435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:25.193305969 +0000 UTC m=+385.126834626" watchObservedRunningTime="2026-11-03 07:05:25.195127544 +0000 UTC m=+385.128656205" Nov 03 07:05:25.951407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:25.951124 4925 generic.go:356] "Generic (PLEG): container finished" podID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerID="32b62f8175b2f7926e3869d64b2159cb3307e75ab857fafc4b3bb73bffe90017" exitCode=0 Nov 03 07:05:25.951880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:25.951638 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" event={"ID":"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9","Type":"ContainerDied","Data":"32b62f8175b2f7926e3869d64b2159cb3307e75ab857fafc4b3bb73bffe90017"} Nov 03 07:05:25.956053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:25.955355 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" event={"ID":"f563e399-e023-42fe-8fc7-4bb2a3d89153","Type":"ContainerStarted","Data":"1609a0ddb46b23d099fa7eb6a73d9a1b16ba0c011efb8c03a18a9229964c84ec"} Nov 03 07:05:25.959214 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:25.959180 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" event={"ID":"9232d701-19e3-47df-9631-9af0a66b6572","Type":"ContainerStarted","Data":"7d2d12a2f641561171b7ac12919d87573a45995b9cf7f30597c3a753c3526b61"} Nov 03 07:05:25.970392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:25.970299 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" event={"ID":"63a2ed70-0fc9-4a8e-8971-49e50a885fd2","Type":"ContainerStarted","Data":"592d8bea876bab2ab11e2dd7b05c42cf84c3186e127e641c1e38e7abd9947db8"} Nov 03 07:05:26.110968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.110943 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:05:26.134156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.134121 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-kube-api-access-5j2tg\" (UniqueName: \"kubernetes.io/projected/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-kube-api-access-5j2tg\") pod \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " Nov 03 07:05:26.134305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.134181 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-bundle\") pod \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " Nov 03 07:05:26.134305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.134214 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-util\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-util\") pod \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\" (UID: \"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e\") " Nov 03 07:05:26.134943 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.134901 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-bundle" pod "6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" (UID: "6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:26.137367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.137232 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-kube-api-access-5j2tg" pod "6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" (UID: "6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e"). InnerVolumeSpecName "kube-api-access-5j2tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:26.141878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.141852 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-util" pod "6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" (UID: "6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:26.233306 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.233242 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" podStartSLOduration=2.894468762 podStartE2EDuration="35.233225966s" totalImagesPullingTime="32.338757204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:26.232243611 +0000 UTC m=+386.165772269" watchObservedRunningTime="2026-11-03 07:05:26.233225966 +0000 UTC m=+386.166754622" Nov 03 07:05:26.235765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.235737 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-util\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:26.235958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.235768 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j2tg\" (UniqueName: \"kubernetes.io/projected/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-kube-api-access-5j2tg\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:26.235958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.235784 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e-bundle\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:26.357278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.357167 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" podStartSLOduration=3.25202058 podStartE2EDuration="35.357146571s" totalImagesPullingTime="32.105125991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:26.356716262 +0000 UTC m=+386.290244918" watchObservedRunningTime="2026-11-03 07:05:26.357146571 +0000 UTC m=+386.290675228" Nov 03 07:05:26.480498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.480444 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" podStartSLOduration=3.560136895 podStartE2EDuration="35.480424947s" totalImagesPullingTime="31.920288052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:26.479835605 +0000 UTC m=+386.413364262" watchObservedRunningTime="2026-11-03 07:05:26.480424947 +0000 UTC m=+386.413953606" Nov 03 07:05:26.604380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.604309 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" podStartSLOduration=3.2917591010000002 podStartE2EDuration="35.604293002s" totalImagesPullingTime="32.312533901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:26.602954123 +0000 UTC m=+386.536482780" watchObservedRunningTime="2026-11-03 07:05:26.604293002 +0000 UTC m=+386.537821658" Nov 03 07:05:26.982891 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.982854 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" event={"ID":"43851410-de5e-4d41-8ff3-f2601a30dd86","Type":"ContainerStarted","Data":"124a0c3347a5ac221c922ad3eea197d7045659944ac7c9c1ed363d94b2acfd10"} Nov 03 07:05:26.985060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.985033 4925 generic.go:356] "Generic (PLEG): container finished" podID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerID="4d9b1f76820a217f841d5fdc99856298cdb0d4c008a4ccb06cbc7d57c581bbf5" exitCode=0 Nov 03 07:05:26.985202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.985103 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" event={"ID":"18dc22e4-c6e6-4055-a715-c6a59340e140","Type":"ContainerDied","Data":"4d9b1f76820a217f841d5fdc99856298cdb0d4c008a4ccb06cbc7d57c581bbf5"} Nov 03 07:05:26.986966 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.986943 4925 generic.go:356] "Generic (PLEG): container finished" podID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerID="592d8bea876bab2ab11e2dd7b05c42cf84c3186e127e641c1e38e7abd9947db8" exitCode=0 Nov 03 07:05:26.987064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.987038 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" event={"ID":"63a2ed70-0fc9-4a8e-8971-49e50a885fd2","Type":"ContainerDied","Data":"592d8bea876bab2ab11e2dd7b05c42cf84c3186e127e641c1e38e7abd9947db8"} Nov 03 07:05:26.988868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.988840 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" event={"ID":"6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e","Type":"ContainerDied","Data":"a4e923e234371576d895d4b7f05e5eb26a1160e5970f695d8a9f57289e9c9cd7"} Nov 03 07:05:26.989033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.988887 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e923e234371576d895d4b7f05e5eb26a1160e5970f695d8a9f57289e9c9cd7" Nov 03 07:05:26.989033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.988857 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lqk5t" Nov 03 07:05:26.991130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.991088 4925 generic.go:356] "Generic (PLEG): container finished" podID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerID="1609a0ddb46b23d099fa7eb6a73d9a1b16ba0c011efb8c03a18a9229964c84ec" exitCode=0 Nov 03 07:05:26.991228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.991132 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" event={"ID":"f563e399-e023-42fe-8fc7-4bb2a3d89153","Type":"ContainerDied","Data":"1609a0ddb46b23d099fa7eb6a73d9a1b16ba0c011efb8c03a18a9229964c84ec"} Nov 03 07:05:26.993126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.993107 4925 generic.go:356] "Generic (PLEG): container finished" podID="9232d701-19e3-47df-9631-9af0a66b6572" containerID="7d2d12a2f641561171b7ac12919d87573a45995b9cf7f30597c3a753c3526b61" exitCode=0 Nov 03 07:05:26.993223 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:26.993170 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" event={"ID":"9232d701-19e3-47df-9631-9af0a66b6572","Type":"ContainerDied","Data":"7d2d12a2f641561171b7ac12919d87573a45995b9cf7f30597c3a753c3526b61"} Nov 03 07:05:27.105370 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.105324 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" podStartSLOduration=2.018060003 podStartE2EDuration="34.105311303s" totalImagesPullingTime="32.0872513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:27.104530703 +0000 UTC m=+387.038059358" watchObservedRunningTime="2026-11-03 07:05:27.105311303 +0000 UTC m=+387.038839958" Nov 03 07:05:27.121226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.121203 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:05:27.142788 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.142761 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-kube-api-access-44xrj\" (UniqueName: \"kubernetes.io/projected/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-kube-api-access-44xrj\") pod \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " Nov 03 07:05:27.142919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.142806 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-bundle\") pod \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " Nov 03 07:05:27.142919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.142865 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-util\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-util\") pod \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\" (UID: \"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9\") " Nov 03 07:05:27.143570 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.143493 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-bundle" pod "3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" (UID: "3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:27.145250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.145223 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-kube-api-access-44xrj" pod "3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" (UID: "3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9"). InnerVolumeSpecName "kube-api-access-44xrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:27.146803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.146778 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-util" pod "3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" (UID: "3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:27.244094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.244062 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44xrj\" (UniqueName: \"kubernetes.io/projected/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-kube-api-access-44xrj\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:27.244094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.244090 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-bundle\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:27.244094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.244100 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9-util\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:27.999732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.999694 4925 generic.go:356] "Generic (PLEG): container finished" podID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerID="124a0c3347a5ac221c922ad3eea197d7045659944ac7c9c1ed363d94b2acfd10" exitCode=0 Nov 03 07:05:27.999732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:27.999726 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" event={"ID":"43851410-de5e-4d41-8ff3-f2601a30dd86","Type":"ContainerDied","Data":"124a0c3347a5ac221c922ad3eea197d7045659944ac7c9c1ed363d94b2acfd10"} Nov 03 07:05:28.001965 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.001945 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" Nov 03 07:05:28.002112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.001990 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5bq6g" event={"ID":"3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9","Type":"ContainerDied","Data":"d8691ee95935b609063d4abfae76636493502b963e368877a5edfa3d4b7625ff"} Nov 03 07:05:28.002112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.002067 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8691ee95935b609063d4abfae76636493502b963e368877a5edfa3d4b7625ff" Nov 03 07:05:28.145273 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.145245 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:05:28.152081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.152053 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-bundle\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-bundle\") pod \"18dc22e4-c6e6-4055-a715-c6a59340e140\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " Nov 03 07:05:28.152216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.152113 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18dc22e4-c6e6-4055-a715-c6a59340e140-kube-api-access-kq6rr\" (UniqueName: \"kubernetes.io/projected/18dc22e4-c6e6-4055-a715-c6a59340e140-kube-api-access-kq6rr\") pod \"18dc22e4-c6e6-4055-a715-c6a59340e140\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " Nov 03 07:05:28.152216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.152194 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-util\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-util\") pod \"18dc22e4-c6e6-4055-a715-c6a59340e140\" (UID: \"18dc22e4-c6e6-4055-a715-c6a59340e140\") " Nov 03 07:05:28.152645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.152615 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-bundle" pod "18dc22e4-c6e6-4055-a715-c6a59340e140" (UID: "18dc22e4-c6e6-4055-a715-c6a59340e140"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.157151 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.157114 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18dc22e4-c6e6-4055-a715-c6a59340e140-kube-api-access-kq6rr" pod "18dc22e4-c6e6-4055-a715-c6a59340e140" (UID: "18dc22e4-c6e6-4055-a715-c6a59340e140"). InnerVolumeSpecName "kube-api-access-kq6rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:28.159923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.159893 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-util" pod "18dc22e4-c6e6-4055-a715-c6a59340e140" (UID: "18dc22e4-c6e6-4055-a715-c6a59340e140"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.176518 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.176495 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:05:28.195124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.195100 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:05:28.209390 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.209367 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:05:28.252902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.252880 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-bundle\") pod \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " Nov 03 07:05:28.253050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.252915 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-bundle\") pod \"9232d701-19e3-47df-9631-9af0a66b6572\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " Nov 03 07:05:28.253050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.252941 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-util\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-util\") pod \"9232d701-19e3-47df-9631-9af0a66b6572\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " Nov 03 07:05:28.253050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.252976 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-kube-api-access-zrgtr\" (UniqueName: \"kubernetes.io/projected/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-kube-api-access-zrgtr\") pod \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " Nov 03 07:05:28.253050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.252997 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-util\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-util\") pod \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\" (UID: \"63a2ed70-0fc9-4a8e-8971-49e50a885fd2\") " Nov 03 07:05:28.253050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253037 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f563e399-e023-42fe-8fc7-4bb2a3d89153-kube-api-access-rpkm9\" (UniqueName: \"kubernetes.io/projected/f563e399-e023-42fe-8fc7-4bb2a3d89153-kube-api-access-rpkm9\") pod \"f563e399-e023-42fe-8fc7-4bb2a3d89153\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " Nov 03 07:05:28.253367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253076 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-util\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-util\") pod \"f563e399-e023-42fe-8fc7-4bb2a3d89153\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " Nov 03 07:05:28.253367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253103 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-bundle\") pod \"f563e399-e023-42fe-8fc7-4bb2a3d89153\" (UID: \"f563e399-e023-42fe-8fc7-4bb2a3d89153\") " Nov 03 07:05:28.253367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253190 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9232d701-19e3-47df-9631-9af0a66b6572-kube-api-access-xhdlp\" (UniqueName: \"kubernetes.io/projected/9232d701-19e3-47df-9631-9af0a66b6572-kube-api-access-xhdlp\") pod \"9232d701-19e3-47df-9631-9af0a66b6572\" (UID: \"9232d701-19e3-47df-9631-9af0a66b6572\") " Nov 03 07:05:28.253524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253440 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-util\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.253524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253456 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18dc22e4-c6e6-4055-a715-c6a59340e140-bundle\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.253524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253469 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6rr\" (UniqueName: \"kubernetes.io/projected/18dc22e4-c6e6-4055-a715-c6a59340e140-kube-api-access-kq6rr\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.253524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253507 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-bundle" pod "9232d701-19e3-47df-9631-9af0a66b6572" (UID: "9232d701-19e3-47df-9631-9af0a66b6572"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.254312 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.253972 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-bundle" pod "f563e399-e023-42fe-8fc7-4bb2a3d89153" (UID: "f563e399-e023-42fe-8fc7-4bb2a3d89153"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.254312 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.254060 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-bundle" pod "63a2ed70-0fc9-4a8e-8971-49e50a885fd2" (UID: "63a2ed70-0fc9-4a8e-8971-49e50a885fd2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.255976 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.255951 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f563e399-e023-42fe-8fc7-4bb2a3d89153-kube-api-access-rpkm9" pod "f563e399-e023-42fe-8fc7-4bb2a3d89153" (UID: "f563e399-e023-42fe-8fc7-4bb2a3d89153"). InnerVolumeSpecName "kube-api-access-rpkm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:28.256259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.256238 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-kube-api-access-zrgtr" pod "63a2ed70-0fc9-4a8e-8971-49e50a885fd2" (UID: "63a2ed70-0fc9-4a8e-8971-49e50a885fd2"). InnerVolumeSpecName "kube-api-access-zrgtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:28.256460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.256439 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9232d701-19e3-47df-9631-9af0a66b6572-kube-api-access-xhdlp" pod "9232d701-19e3-47df-9631-9af0a66b6572" (UID: "9232d701-19e3-47df-9631-9af0a66b6572"). InnerVolumeSpecName "kube-api-access-xhdlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:28.261078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.261055 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-util" pod "63a2ed70-0fc9-4a8e-8971-49e50a885fd2" (UID: "63a2ed70-0fc9-4a8e-8971-49e50a885fd2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.261651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.261629 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-util" pod "9232d701-19e3-47df-9631-9af0a66b6572" (UID: "9232d701-19e3-47df-9631-9af0a66b6572"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.262995 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.262969 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-util" pod "f563e399-e023-42fe-8fc7-4bb2a3d89153" (UID: "f563e399-e023-42fe-8fc7-4bb2a3d89153"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:28.354403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354370 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-util\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354396 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpkm9\" (UniqueName: \"kubernetes.io/projected/f563e399-e023-42fe-8fc7-4bb2a3d89153-kube-api-access-rpkm9\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354407 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-util\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354415 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f563e399-e023-42fe-8fc7-4bb2a3d89153-bundle\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354424 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhdlp\" (UniqueName: \"kubernetes.io/projected/9232d701-19e3-47df-9631-9af0a66b6572-kube-api-access-xhdlp\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354432 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-bundle\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354440 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-bundle\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354447 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9232d701-19e3-47df-9631-9af0a66b6572-util\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.354618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.354455 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrgtr\" (UniqueName: \"kubernetes.io/projected/63a2ed70-0fc9-4a8e-8971-49e50a885fd2-kube-api-access-zrgtr\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:28.389551 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.389469 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:28.684927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.684801 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:28.907500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:28.907462 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:29.008556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.008477 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" Nov 03 07:05:29.008556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.008488 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acvhr7" event={"ID":"63a2ed70-0fc9-4a8e-8971-49e50a885fd2","Type":"ContainerDied","Data":"cbe5c63fef988bdf80ab72675933b47ecb50dc16504ab4e7d89ae1c5acb9cfc9"} Nov 03 07:05:29.008556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.008533 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe5c63fef988bdf80ab72675933b47ecb50dc16504ab4e7d89ae1c5acb9cfc9" Nov 03 07:05:29.016633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.016607 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" Nov 03 07:05:29.016768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.016606 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l72qw" event={"ID":"f563e399-e023-42fe-8fc7-4bb2a3d89153","Type":"ContainerDied","Data":"b189612d7931f82f90ee67b35c58574bde8b3b31cb3c63d9c140e6861f0a025b"} Nov 03 07:05:29.016768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.016764 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b189612d7931f82f90ee67b35c58574bde8b3b31cb3c63d9c140e6861f0a025b" Nov 03 07:05:29.018700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.018683 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" Nov 03 07:05:29.018817 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.018708 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq5mxx" event={"ID":"9232d701-19e3-47df-9631-9af0a66b6572","Type":"ContainerDied","Data":"959047cd39a86fe0609d74346667d96c33189d2ca1ee63f437fd1e38e9e5560f"} Nov 03 07:05:29.018817 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.018753 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="959047cd39a86fe0609d74346667d96c33189d2ca1ee63f437fd1e38e9e5560f" Nov 03 07:05:29.020855 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.020823 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" event={"ID":"18dc22e4-c6e6-4055-a715-c6a59340e140","Type":"ContainerDied","Data":"22d2a951b0670cd5adf6e9d6eff34f7e5aba627e389957b313adf84cc0aa6f6c"} Nov 03 07:05:29.020965 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.020868 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22d2a951b0670cd5adf6e9d6eff34f7e5aba627e389957b313adf84cc0aa6f6c" Nov 03 07:05:29.020965 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.020872 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht46" Nov 03 07:05:29.147337 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.147315 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:05:29.160254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.160233 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-util\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-util\") pod \"43851410-de5e-4d41-8ff3-f2601a30dd86\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " Nov 03 07:05:29.160351 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.160268 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43851410-de5e-4d41-8ff3-f2601a30dd86-kube-api-access-4n5g2\" (UniqueName: \"kubernetes.io/projected/43851410-de5e-4d41-8ff3-f2601a30dd86-kube-api-access-4n5g2\") pod \"43851410-de5e-4d41-8ff3-f2601a30dd86\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " Nov 03 07:05:29.160351 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.160292 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-bundle\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-bundle\") pod \"43851410-de5e-4d41-8ff3-f2601a30dd86\" (UID: \"43851410-de5e-4d41-8ff3-f2601a30dd86\") " Nov 03 07:05:29.160683 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.160657 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-bundle" pod "43851410-de5e-4d41-8ff3-f2601a30dd86" (UID: "43851410-de5e-4d41-8ff3-f2601a30dd86"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:29.162482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.162458 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43851410-de5e-4d41-8ff3-f2601a30dd86-kube-api-access-4n5g2" pod "43851410-de5e-4d41-8ff3-f2601a30dd86" (UID: "43851410-de5e-4d41-8ff3-f2601a30dd86"). InnerVolumeSpecName "kube-api-access-4n5g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:29.167694 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.167667 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-util" pod "43851410-de5e-4d41-8ff3-f2601a30dd86" (UID: "43851410-de5e-4d41-8ff3-f2601a30dd86"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:29.242999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.242967 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:29.261027 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.260927 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-util\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:29.261027 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.260957 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5g2\" (UniqueName: \"kubernetes.io/projected/43851410-de5e-4d41-8ff3-f2601a30dd86-kube-api-access-4n5g2\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:29.261027 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.260974 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43851410-de5e-4d41-8ff3-f2601a30dd86-bundle\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:29.355384 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355351 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4"] Nov 03 07:05:29.355651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355639 4925 cpu_manager.go:414] "containerMap: removing container" podUID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerName="util" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355653 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerName="util" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355664 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9232d701-19e3-47df-9631-9af0a66b6572" containerName="pull" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355669 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9232d701-19e3-47df-9631-9af0a66b6572" containerName="pull" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355675 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerName="extract" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355680 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerName="extract" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355687 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerName="extract" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355691 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerName="extract" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355698 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerName="util" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355703 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerName="util" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355710 4925 cpu_manager.go:414] "containerMap: removing container" podUID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerName="pull" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355714 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerName="pull" Nov 03 07:05:29.355716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355720 4925 cpu_manager.go:414] "containerMap: removing container" podUID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355724 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355730 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355733 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355742 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9232d701-19e3-47df-9631-9af0a66b6572" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355746 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9232d701-19e3-47df-9631-9af0a66b6572" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355752 4925 cpu_manager.go:414] "containerMap: removing container" podUID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355757 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355762 4925 cpu_manager.go:414] "containerMap: removing container" podUID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355766 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355775 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355779 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355785 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355789 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355797 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355800 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355809 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355812 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355819 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355823 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355828 4925 cpu_manager.go:414] "containerMap: removing container" podUID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355832 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerName="pull" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355837 4925 cpu_manager.go:414] "containerMap: removing container" podUID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355841 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355849 4925 cpu_manager.go:414] "containerMap: removing container" podUID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355853 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerName="util" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355861 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9232d701-19e3-47df-9631-9af0a66b6572" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355865 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9232d701-19e3-47df-9631-9af0a66b6572" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355871 4925 cpu_manager.go:414] "containerMap: removing container" podUID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355875 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355919 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f563e399-e023-42fe-8fc7-4bb2a3d89153" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355927 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="18dc22e4-c6e6-4055-a715-c6a59340e140" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355934 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="43851410-de5e-4d41-8ff3-f2601a30dd86" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355942 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9232d701-19e3-47df-9631-9af0a66b6572" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355948 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a2ed70-0fc9-4a8e-8971-49e50a885fd2" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355954 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9ce6c0-02e1-4c08-85d3-6a45ed63df6e" containerName="extract" Nov 03 07:05:29.356073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.355960 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6030cf-b2ed-46d7-9c7e-c75d386fe4a9" containerName="extract" Nov 03 07:05:29.357804 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.357789 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.413459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.413433 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 07:05:29.413459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.413439 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 07:05:29.413615 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.413502 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 07:05:29.413814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.413799 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-frrnv\"" Nov 03 07:05:29.413865 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.413834 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 07:05:29.413919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.413863 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 07:05:29.463170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.463141 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.463299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.463174 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.463299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.463212 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbcee74e-9656-4aa2-a8bb-18f2db697efd-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.463400 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.463316 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbcee74e-9656-4aa2-a8bb-18f2db697efd-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.463400 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.463349 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbcz\" (UniqueName: \"kubernetes.io/projected/cbcee74e-9656-4aa2-a8bb-18f2db697efd-kube-api-access-5vbcz\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.463510 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.463427 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.497859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.497825 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4"] Nov 03 07:05:29.499695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.499661 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:29.564339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.564315 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.564503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.564347 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.564503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.564387 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbcee74e-9656-4aa2-a8bb-18f2db697efd-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.564503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.564410 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbcee74e-9656-4aa2-a8bb-18f2db697efd-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.564503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.564431 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbcz\" (UniqueName: \"kubernetes.io/projected/cbcee74e-9656-4aa2-a8bb-18f2db697efd-kube-api-access-5vbcz\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.564503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.564453 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.564901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.564868 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbcee74e-9656-4aa2-a8bb-18f2db697efd-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.566988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.566968 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.567108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.566997 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.567268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.567251 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbcee74e-9656-4aa2-a8bb-18f2db697efd-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.567396 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.567379 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbcee74e-9656-4aa2-a8bb-18f2db697efd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.633054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.633000 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbcz\" (UniqueName: \"kubernetes.io/projected/cbcee74e-9656-4aa2-a8bb-18f2db697efd-kube-api-access-5vbcz\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4\" (UID: \"cbcee74e-9656-4aa2-a8bb-18f2db697efd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.670038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.669997 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:29.717275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.717241 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:29.939046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:29.938948 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:30.027061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:30.027033 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" Nov 03 07:05:30.027061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:30.027051 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448hhvh" event={"ID":"43851410-de5e-4d41-8ff3-f2601a30dd86","Type":"ContainerDied","Data":"c4f29a9cad3d5248b818a0657096c6cae8550a06609fcf65d445f7d139e07736"} Nov 03 07:05:30.027446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:30.027093 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f29a9cad3d5248b818a0657096c6cae8550a06609fcf65d445f7d139e07736" Nov 03 07:05:30.170851 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:05:30.170825 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcee74e_9656_4aa2_a8bb_18f2db697efd.slice/crio-af85876df91f133aa90e81a6458de7e4c1982bc3a3e6fc1edaf117e66cf2b984 WatchSource:0}: Error finding container af85876df91f133aa90e81a6458de7e4c1982bc3a3e6fc1edaf117e66cf2b984: Status 404 returned error can't find the container with id af85876df91f133aa90e81a6458de7e4c1982bc3a3e6fc1edaf117e66cf2b984 Nov 03 07:05:30.171822 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:30.171797 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4"] Nov 03 07:05:31.033034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:31.032982 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" event={"ID":"cbcee74e-9656-4aa2-a8bb-18f2db697efd","Type":"ContainerStarted","Data":"af85876df91f133aa90e81a6458de7e4c1982bc3a3e6fc1edaf117e66cf2b984"} Nov 03 07:05:32.229986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:32.229946 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:32.474967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:32.474930 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:34.583314 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:34.583284 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:35.053529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:35.053499 4925 generic.go:356] "Generic (PLEG): container finished" podID="cbcee74e-9656-4aa2-a8bb-18f2db697efd" containerID="3fd797f4a7c8d894f6529f88892a98b139c4b8980d1939a8be5904479b8b9965" exitCode=0 Nov 03 07:05:35.053674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:35.053585 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" event={"ID":"cbcee74e-9656-4aa2-a8bb-18f2db697efd","Type":"ContainerDied","Data":"3fd797f4a7c8d894f6529f88892a98b139c4b8980d1939a8be5904479b8b9965"} Nov 03 07:05:36.062959 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:36.062936 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" event={"ID":"cbcee74e-9656-4aa2-a8bb-18f2db697efd","Type":"ContainerStarted","Data":"980ede84e1fbfa874ee39bef864198092de3d15b6b9574a6be96fd0c306209fc"} Nov 03 07:05:36.063369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:36.063143 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:36.217668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:36.217608 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" podStartSLOduration=2.139677754 podStartE2EDuration="7.217595585s" totalImagesPullingTime="4.590163643s" totalInitContainerRuntime="487.754188ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:36.21628806 +0000 UTC m=+396.149816728" watchObservedRunningTime="2026-11-03 07:05:36.217595585 +0000 UTC m=+396.151124241" Nov 03 07:05:37.242952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:37.242912 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:37.440642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:37.440608 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:37.901642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:37.901606 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv"] Nov 03 07:05:37.903608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:37.903592 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:37.936175 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:37.936149 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afc9e2a7-f066-4a57-add2-93483ae9e045-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv\" (UID: \"afc9e2a7-f066-4a57-add2-93483ae9e045\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:37.936292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:37.936181 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7psgb\" (UniqueName: \"kubernetes.io/projected/afc9e2a7-f066-4a57-add2-93483ae9e045-kube-api-access-7psgb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv\" (UID: \"afc9e2a7-f066-4a57-add2-93483ae9e045\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:38.014963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.014933 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 07:05:38.015104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.014967 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k5tf8\"" Nov 03 07:05:38.015104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.015042 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 07:05:38.016712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.016692 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv"] Nov 03 07:05:38.037140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.037117 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7psgb\" (UniqueName: \"kubernetes.io/projected/afc9e2a7-f066-4a57-add2-93483ae9e045-kube-api-access-7psgb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv\" (UID: \"afc9e2a7-f066-4a57-add2-93483ae9e045\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:38.037256 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.037214 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afc9e2a7-f066-4a57-add2-93483ae9e045-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv\" (UID: \"afc9e2a7-f066-4a57-add2-93483ae9e045\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:38.037646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.037625 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afc9e2a7-f066-4a57-add2-93483ae9e045-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv\" (UID: \"afc9e2a7-f066-4a57-add2-93483ae9e045\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:38.109627 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.109603 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7psgb\" (UniqueName: \"kubernetes.io/projected/afc9e2a7-f066-4a57-add2-93483ae9e045-kube-api-access-7psgb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv\" (UID: \"afc9e2a7-f066-4a57-add2-93483ae9e045\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:38.213616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.213538 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" Nov 03 07:05:38.692068 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.692044 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv"] Nov 03 07:05:38.692923 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:05:38.692884 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc9e2a7_f066_4a57_add2_93483ae9e045.slice/crio-54b25182b7bdf60f9d177409c1324d1289caeade006b086f74e7b5f2572ce96e WatchSource:0}: Error finding container 54b25182b7bdf60f9d177409c1324d1289caeade006b086f74e7b5f2572ce96e: Status 404 returned error can't find the container with id 54b25182b7bdf60f9d177409c1324d1289caeade006b086f74e7b5f2572ce96e Nov 03 07:05:38.876146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:38.876112 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:39.077988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:39.077956 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" event={"ID":"afc9e2a7-f066-4a57-add2-93483ae9e045","Type":"ContainerStarted","Data":"54b25182b7bdf60f9d177409c1324d1289caeade006b086f74e7b5f2572ce96e"} Nov 03 07:05:39.135704 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:05:39.135669 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:05:40.208001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:40.207964 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:40.450153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:40.450064 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:41.189548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:41.189517 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:42.093898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:42.093860 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" event={"ID":"afc9e2a7-f066-4a57-add2-93483ae9e045","Type":"ContainerStarted","Data":"b16ebdda9ea39ea4df8defee5a1d6f3b68415ff98bca1d81023eb8c6261a40eb"} Nov 03 07:05:42.326148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:42.326097 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lnsdv" podStartSLOduration=2.871482876 podStartE2EDuration="5.326083596s" totalImagesPullingTime="2.45460072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:42.273594089 +0000 UTC m=+402.207122744" watchObservedRunningTime="2026-11-03 07:05:42.326083596 +0000 UTC m=+402.259612251" Nov 03 07:05:42.385914 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:42.385829 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:42.790690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:42.790596 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:43.048218 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:43.048126 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:45.245567 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.245528 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:45.577474 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.577441 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt"] Nov 03 07:05:45.581233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.581212 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:45.641043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.641005 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt"] Nov 03 07:05:45.686953 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.686930 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 07:05:45.687114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.687101 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 07:05:45.687160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.687136 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-66t8t\"" Nov 03 07:05:45.698625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.698592 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ea6444c-a40f-42a5-84a2-cb113ea168a0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fl9vt\" (UID: \"7ea6444c-a40f-42a5-84a2-cb113ea168a0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:45.698775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.698641 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfqj\" (UniqueName: \"kubernetes.io/projected/7ea6444c-a40f-42a5-84a2-cb113ea168a0-kube-api-access-pjfqj\") pod \"cert-manager-cainjector-5c57bbd4fc-fl9vt\" (UID: \"7ea6444c-a40f-42a5-84a2-cb113ea168a0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:45.742183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.742149 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:45.799951 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.799920 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ea6444c-a40f-42a5-84a2-cb113ea168a0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fl9vt\" (UID: \"7ea6444c-a40f-42a5-84a2-cb113ea168a0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:45.800091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.799959 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfqj\" (UniqueName: \"kubernetes.io/projected/7ea6444c-a40f-42a5-84a2-cb113ea168a0-kube-api-access-pjfqj\") pod \"cert-manager-cainjector-5c57bbd4fc-fl9vt\" (UID: \"7ea6444c-a40f-42a5-84a2-cb113ea168a0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:45.864633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.864566 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ea6444c-a40f-42a5-84a2-cb113ea168a0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fl9vt\" (UID: \"7ea6444c-a40f-42a5-84a2-cb113ea168a0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:45.864733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.864651 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfqj\" (UniqueName: \"kubernetes.io/projected/7ea6444c-a40f-42a5-84a2-cb113ea168a0-kube-api-access-pjfqj\") pod \"cert-manager-cainjector-5c57bbd4fc-fl9vt\" (UID: \"7ea6444c-a40f-42a5-84a2-cb113ea168a0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:45.906190 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:45.906164 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" Nov 03 07:05:46.286609 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.286585 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt"] Nov 03 07:05:46.288353 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:05:46.288304 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ea6444c_a40f_42a5_84a2_cb113ea168a0.slice/crio-4c486effc5739e920851cc7607b0e808aee13b48884985f4b20701fd06758b21 WatchSource:0}: Error finding container 4c486effc5739e920851cc7607b0e808aee13b48884985f4b20701fd06758b21: Status 404 returned error can't find the container with id 4c486effc5739e920851cc7607b0e808aee13b48884985f4b20701fd06758b21 Nov 03 07:05:46.394340 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.394314 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-884dw"] Nov 03 07:05:46.398694 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.398675 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" Nov 03 07:05:46.454380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.454355 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 07:05:46.454380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.454371 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 07:05:46.454543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.454386 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tzjvg\"" Nov 03 07:05:46.506305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.506276 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmlck\" (UniqueName: \"kubernetes.io/projected/a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd-kube-api-access-fmlck\") pod \"authorino-operator-7b7bb4bc77-884dw\" (UID: \"a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" Nov 03 07:05:46.506877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.506856 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-884dw"] Nov 03 07:05:46.607256 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.607224 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmlck\" (UniqueName: \"kubernetes.io/projected/a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd-kube-api-access-fmlck\") pod \"authorino-operator-7b7bb4bc77-884dw\" (UID: \"a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" Nov 03 07:05:46.677392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.677363 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmlck\" (UniqueName: \"kubernetes.io/projected/a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd-kube-api-access-fmlck\") pod \"authorino-operator-7b7bb4bc77-884dw\" (UID: \"a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" Nov 03 07:05:46.711913 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:46.711882 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" Nov 03 07:05:47.070187 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:47.070157 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-rrzm4" Nov 03 07:05:47.086055 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:47.086033 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-884dw"] Nov 03 07:05:47.087911 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:05:47.087892 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c4f39f_4f97_4dfa_9aaa_d24f861a97cd.slice/crio-2284111c315ea783f31cd07f77416f4676065095d331c1c4637b6a9743a6dd54 WatchSource:0}: Error finding container 2284111c315ea783f31cd07f77416f4676065095d331c1c4637b6a9743a6dd54: Status 404 returned error can't find the container with id 2284111c315ea783f31cd07f77416f4676065095d331c1c4637b6a9743a6dd54 Nov 03 07:05:47.117029 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:47.116993 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" event={"ID":"a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd","Type":"ContainerStarted","Data":"2284111c315ea783f31cd07f77416f4676065095d331c1c4637b6a9743a6dd54"} Nov 03 07:05:47.118125 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:47.118102 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" event={"ID":"7ea6444c-a40f-42a5-84a2-cb113ea168a0","Type":"ContainerStarted","Data":"4c486effc5739e920851cc7607b0e808aee13b48884985f4b20701fd06758b21"} Nov 03 07:05:47.975662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:47.975612 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:48.839513 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:48.839467 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:49.089997 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:49.089801 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:50.017240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:50.017208 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:50.138980 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:50.138942 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" event={"ID":"7ea6444c-a40f-42a5-84a2-cb113ea168a0","Type":"ContainerStarted","Data":"2559d5f591594db4afdae1420a55edce8e7e7db664b8d2fa82ddd9c5613e1ed7"} Nov 03 07:05:50.276394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:50.276258 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:51.018324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.018226 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fl9vt" podStartSLOduration=2.296309711 podStartE2EDuration="6.018210975s" totalImagesPullingTime="3.721901264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:50.260793224 +0000 UTC m=+410.194321873" watchObservedRunningTime="2026-11-03 07:05:51.018210975 +0000 UTC m=+410.951739630" Nov 03 07:05:51.018475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.018362 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2"] Nov 03 07:05:51.021806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.021790 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" Nov 03 07:05:51.082599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.082573 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-td48k\"" Nov 03 07:05:51.133797 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.133766 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2"] Nov 03 07:05:51.145457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.145428 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" event={"ID":"a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd","Type":"ContainerStarted","Data":"adaf85ef72d0814fa0a5d57d209a52a3ba8d829f69e8fdcffa724997dc69a365"} Nov 03 07:05:51.145872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.145662 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" Nov 03 07:05:51.145981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.145962 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlvq\" (UniqueName: \"kubernetes.io/projected/22b735f0-6048-48a9-aebf-bf4f7270ab02-kube-api-access-cvlvq\") pod \"limitador-operator-controller-manager-5778bb4449-nsnp2\" (UID: \"22b735f0-6048-48a9-aebf-bf4f7270ab02\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" Nov 03 07:05:51.246814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.246773 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlvq\" (UniqueName: \"kubernetes.io/projected/22b735f0-6048-48a9-aebf-bf4f7270ab02-kube-api-access-cvlvq\") pod \"limitador-operator-controller-manager-5778bb4449-nsnp2\" (UID: \"22b735f0-6048-48a9-aebf-bf4f7270ab02\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" Nov 03 07:05:51.282035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.281925 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" podStartSLOduration=1.803604448 podStartE2EDuration="5.281911344s" totalImagesPullingTime="3.478306896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:51.280501124 +0000 UTC m=+411.214029796" watchObservedRunningTime="2026-11-03 07:05:51.281911344 +0000 UTC m=+411.215440000" Nov 03 07:05:51.317713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.317679 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlvq\" (UniqueName: \"kubernetes.io/projected/22b735f0-6048-48a9-aebf-bf4f7270ab02-kube-api-access-cvlvq\") pod \"limitador-operator-controller-manager-5778bb4449-nsnp2\" (UID: \"22b735f0-6048-48a9-aebf-bf4f7270ab02\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" Nov 03 07:05:51.334618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.334592 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" Nov 03 07:05:51.671622 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.671584 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:51.717310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.717276 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2"] Nov 03 07:05:51.718770 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:05:51.718744 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b735f0_6048_48a9_aebf_bf4f7270ab02.slice/crio-56cddd2dd92206d35ee8333bc39776650882cd50d9f4408ddb7d3e74c55d38c3 WatchSource:0}: Error finding container 56cddd2dd92206d35ee8333bc39776650882cd50d9f4408ddb7d3e74c55d38c3: Status 404 returned error can't find the container with id 56cddd2dd92206d35ee8333bc39776650882cd50d9f4408ddb7d3e74c55d38c3 Nov 03 07:05:51.943870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:51.943791 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:52.151039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:52.150991 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" event={"ID":"22b735f0-6048-48a9-aebf-bf4f7270ab02","Type":"ContainerStarted","Data":"56cddd2dd92206d35ee8333bc39776650882cd50d9f4408ddb7d3e74c55d38c3"} Nov 03 07:05:52.309453 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:52.309378 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:52.595678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:52.595644 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:52.814935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:52.814895 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:53.502809 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:53.502770 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:54.162832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:54.162803 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" event={"ID":"22b735f0-6048-48a9-aebf-bf4f7270ab02","Type":"ContainerStarted","Data":"c1bf910209fe77554442f57eb679c5be04ce618e172396a2ad0a35698e2d7ff9"} Nov 03 07:05:54.163153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:54.162935 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" Nov 03 07:05:54.328390 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:54.328338 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" podStartSLOduration=1.965022584 podStartE2EDuration="4.328324253s" totalImagesPullingTime="2.363301669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:54.326375965 +0000 UTC m=+414.259904622" watchObservedRunningTime="2026-11-03 07:05:54.328324253 +0000 UTC m=+414.261852908" Nov 03 07:05:55.320129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.320090 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:55.541927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.541883 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:55.710188 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.710113 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m"] Nov 03 07:05:55.713563 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.713543 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" Nov 03 07:05:55.787509 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.787476 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcclq\" (UniqueName: \"kubernetes.io/projected/2e8cbf75-be5c-471c-96ee-22721f1b276c-kube-api-access-fcclq\") pod \"dns-operator-controller-manager-8557cbdb84-wwj4m\" (UID: \"2e8cbf75-be5c-471c-96ee-22721f1b276c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" Nov 03 07:05:55.821581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.821546 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 07:05:55.821724 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.821678 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cxsmm\"" Nov 03 07:05:55.823554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.823537 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m"] Nov 03 07:05:55.888945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.888919 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcclq\" (UniqueName: \"kubernetes.io/projected/2e8cbf75-be5c-471c-96ee-22721f1b276c-kube-api-access-fcclq\") pod \"dns-operator-controller-manager-8557cbdb84-wwj4m\" (UID: \"2e8cbf75-be5c-471c-96ee-22721f1b276c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" Nov 03 07:05:55.953380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:55.953348 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcclq\" (UniqueName: \"kubernetes.io/projected/2e8cbf75-be5c-471c-96ee-22721f1b276c-kube-api-access-fcclq\") pod \"dns-operator-controller-manager-8557cbdb84-wwj4m\" (UID: \"2e8cbf75-be5c-471c-96ee-22721f1b276c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" Nov 03 07:05:56.025894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:56.025818 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" Nov 03 07:05:56.383219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:56.383188 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:56.401216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:56.401192 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m"] Nov 03 07:05:56.402879 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:05:56.402853 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8cbf75_be5c_471c_96ee_22721f1b276c.slice/crio-4d05dc7a9d263c1c7c66f807f072f7263c8c9002029a736530c7b26ec96c5f9a WatchSource:0}: Error finding container 4d05dc7a9d263c1c7c66f807f072f7263c8c9002029a736530c7b26ec96c5f9a: Status 404 returned error can't find the container with id 4d05dc7a9d263c1c7c66f807f072f7263c8c9002029a736530c7b26ec96c5f9a Nov 03 07:05:56.630121 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:56.630089 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:56.993550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:56.993517 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:57.178585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:57.178548 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" event={"ID":"2e8cbf75-be5c-471c-96ee-22721f1b276c","Type":"ContainerStarted","Data":"4d05dc7a9d263c1c7c66f807f072f7263c8c9002029a736530c7b26ec96c5f9a"} Nov 03 07:05:57.194510 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:57.194477 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:57.571662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:57.571626 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:57.780332 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:57.780296 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:05:59.196640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:59.196558 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" event={"ID":"2e8cbf75-be5c-471c-96ee-22721f1b276c","Type":"ContainerStarted","Data":"fe5bd6dc4921fe0e7a0caf1e52d5bd267f8eefb20653087cd56e58e76896aff6"} Nov 03 07:05:59.196992 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:59.196692 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" Nov 03 07:05:59.320274 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:05:59.320226 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" podStartSLOduration=1.804334125 podStartE2EDuration="4.32021252s" totalImagesPullingTime="2.515878395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:59.318934046 +0000 UTC m=+419.252462718" watchObservedRunningTime="2026-11-03 07:05:59.32021252 +0000 UTC m=+419.253741175" Nov 03 07:06:01.581279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.581247 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh"] Nov 03 07:06:01.585876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.585859 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:01.645749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.645720 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ckjh4\"" Nov 03 07:06:01.695350 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.695317 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh"] Nov 03 07:06:01.737098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.737070 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6bzgh\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:01.737242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.737125 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhgh\" (UniqueName: \"kubernetes.io/projected/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-kube-api-access-rhhgh\") pod \"kuadrant-operator-controller-manager-678d87766f-6bzgh\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:01.838181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.838105 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6bzgh\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:01.838181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.838151 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhgh\" (UniqueName: \"kubernetes.io/projected/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-kube-api-access-rhhgh\") pod \"kuadrant-operator-controller-manager-678d87766f-6bzgh\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:01.838505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.838483 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6bzgh\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:01.914642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:01.914606 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhgh\" (UniqueName: \"kubernetes.io/projected/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-kube-api-access-rhhgh\") pod \"kuadrant-operator-controller-manager-678d87766f-6bzgh\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:02.153407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:02.153320 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-884dw" Nov 03 07:06:02.198629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:02.198591 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:02.625578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:02.625551 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh"] Nov 03 07:06:02.626473 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:06:02.626446 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b816edd_5ac9_44ec_86f9_f50b6e5dc545.slice/crio-1512484978a7e5be2fc9f29898bd4c31065e4a655661149f374fab2a8a956b96 WatchSource:0}: Error finding container 1512484978a7e5be2fc9f29898bd4c31065e4a655661149f374fab2a8a956b96: Status 404 returned error can't find the container with id 1512484978a7e5be2fc9f29898bd4c31065e4a655661149f374fab2a8a956b96 Nov 03 07:06:03.224566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:03.224527 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" event={"ID":"4b816edd-5ac9-44ec-86f9-f50b6e5dc545","Type":"ContainerStarted","Data":"1512484978a7e5be2fc9f29898bd4c31065e4a655661149f374fab2a8a956b96"} Nov 03 07:06:04.038848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:04.038808 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:04.247500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:04.247457 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:04.472987 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:04.472949 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:05.170723 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:05.170691 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nsnp2" Nov 03 07:06:07.168639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:07.168603 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:07.453035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:07.452923 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:07.819816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:07.819774 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:08.049094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:08.049056 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:08.254396 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:08.254302 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" event={"ID":"4b816edd-5ac9-44ec-86f9-f50b6e5dc545","Type":"ContainerStarted","Data":"36f69426dcca3cb3407f000efcb2dd88392cd06b9a083bf455bdb75701383db8"} Nov 03 07:06:08.254786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:08.254523 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:08.479790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:08.479733 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" podStartSLOduration=2.863573587 podStartE2EDuration="7.479720098s" totalImagesPullingTime="4.616146511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:08.427381618 +0000 UTC m=+428.360910270" watchObservedRunningTime="2026-11-03 07:06:08.479720098 +0000 UTC m=+428.413248755" Nov 03 07:06:09.137388 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:06:09.137352 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:06:09.350798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:09.350756 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:09.604374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:09.604333 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:09.805243 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:09.805208 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:09.912658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:09.912581 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9"] Nov 03 07:06:09.922171 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:09.922130 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:09.980995 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:09.980964 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9"] Nov 03 07:06:10.009159 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.009132 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:10.011258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011226 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh727\" (UniqueName: \"kubernetes.io/projected/8305581d-8cb9-4e1f-b5b5-c84577035c7d-kube-api-access-dh727\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011273 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011305 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011340 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011542 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011402 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011597 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011561 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011608 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011711 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011667 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.011762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.011724 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.028734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.028710 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nslph\"" Nov 03 07:06:10.112909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.112874 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.112933 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113098 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113141 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113204 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113247 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh727\" (UniqueName: \"kubernetes.io/projected/8305581d-8cb9-4e1f-b5b5-c84577035c7d-kube-api-access-dh727\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113277 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113320 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113533 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113347 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113625 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.113944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113915 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.114105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.113962 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.114105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.114003 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.114105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.114069 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.115450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.115431 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.115665 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.115648 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.177366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.177308 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8305581d-8cb9-4e1f-b5b5-c84577035c7d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.177561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.177537 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh727\" (UniqueName: \"kubernetes.io/projected/8305581d-8cb9-4e1f-b5b5-c84577035c7d-kube-api-access-dh727\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9\" (UID: \"8305581d-8cb9-4e1f-b5b5-c84577035c7d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.201905 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.201846 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:10.203159 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.203139 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wwj4m" Nov 03 07:06:10.234868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.234841 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:10.399247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.399160 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:10.669065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.669037 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9"] Nov 03 07:06:10.670672 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:06:10.670648 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8305581d_8cb9_4e1f_b5b5_c84577035c7d.slice/crio-9467ba620db6b2c6a8ce3a849208b66a605ab2d0edd89b6551d6b02f4f4ce750 WatchSource:0}: Error finding container 9467ba620db6b2c6a8ce3a849208b66a605ab2d0edd89b6551d6b02f4f4ce750: Status 404 returned error can't find the container with id 9467ba620db6b2c6a8ce3a849208b66a605ab2d0edd89b6551d6b02f4f4ce750 Nov 03 07:06:10.672670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.672641 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:06:10.672756 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.672707 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:06:10.672756 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:10.672738 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:06:11.269770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:11.269734 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" event={"ID":"8305581d-8cb9-4e1f-b5b5-c84577035c7d","Type":"ContainerStarted","Data":"a67dad8e203cc1cb61b5884725c582280300ab6d4c37f01225d04404a867931e"} Nov 03 07:06:11.269921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:11.269785 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" event={"ID":"8305581d-8cb9-4e1f-b5b5-c84577035c7d","Type":"ContainerStarted","Data":"9467ba620db6b2c6a8ce3a849208b66a605ab2d0edd89b6551d6b02f4f4ce750"} Nov 03 07:06:11.397053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:11.396979 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" podStartSLOduration=2.396965857 podStartE2EDuration="2.396965857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:11.395275531 +0000 UTC m=+431.328804210" watchObservedRunningTime="2026-11-03 07:06:11.396965857 +0000 UTC m=+431.330494570" Nov 03 07:06:12.235469 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:12.235435 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:12.240675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:12.240653 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:12.274898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:12.274873 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:12.275772 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:12.275752 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9" Nov 03 07:06:15.790699 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:15.790660 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:15.998799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:15.998755 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:16.199884 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:16.199853 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:17.644380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:17.644342 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:17.909074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:17.908955 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:18.109768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:18.109735 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:18.303744 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:18.303657 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:19.250319 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.250284 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sfcmr"] Nov 03 07:06:19.253719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.253689 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" Nov 03 07:06:19.261236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.261212 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:06:19.314035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.313989 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sfcmr"] Nov 03 07:06:19.362439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.362407 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zmffl\"" Nov 03 07:06:19.395874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.395839 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst5b\" (UniqueName: \"kubernetes.io/projected/5842ddf7-7844-4027-81e5-93fb95dd9abb-kube-api-access-dst5b\") pod \"authorino-57f7f8659f-sfcmr\" (UID: \"5842ddf7-7844-4027-81e5-93fb95dd9abb\") " pod="kuadrant-system/authorino-57f7f8659f-sfcmr" Nov 03 07:06:19.450889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.450859 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:19.497155 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.497131 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dst5b\" (UniqueName: \"kubernetes.io/projected/5842ddf7-7844-4027-81e5-93fb95dd9abb-kube-api-access-dst5b\") pod \"authorino-57f7f8659f-sfcmr\" (UID: \"5842ddf7-7844-4027-81e5-93fb95dd9abb\") " pod="kuadrant-system/authorino-57f7f8659f-sfcmr" Nov 03 07:06:19.562912 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.562844 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst5b\" (UniqueName: \"kubernetes.io/projected/5842ddf7-7844-4027-81e5-93fb95dd9abb-kube-api-access-dst5b\") pod \"authorino-57f7f8659f-sfcmr\" (UID: \"5842ddf7-7844-4027-81e5-93fb95dd9abb\") " pod="kuadrant-system/authorino-57f7f8659f-sfcmr" Nov 03 07:06:19.566594 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.566574 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" Nov 03 07:06:19.700909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.700877 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:19.941251 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:19.941220 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sfcmr"] Nov 03 07:06:19.942369 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:06:19.942345 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5842ddf7_7844_4027_81e5_93fb95dd9abb.slice/crio-94b37a4b74de645b7b870949adc733d978419c57b68cbbc9764a4506c7530831 WatchSource:0}: Error finding container 94b37a4b74de645b7b870949adc733d978419c57b68cbbc9764a4506c7530831: Status 404 returned error can't find the container with id 94b37a4b74de645b7b870949adc733d978419c57b68cbbc9764a4506c7530831 Nov 03 07:06:20.315795 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:20.315754 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" event={"ID":"5842ddf7-7844-4027-81e5-93fb95dd9abb","Type":"ContainerStarted","Data":"94b37a4b74de645b7b870949adc733d978419c57b68cbbc9764a4506c7530831"} Nov 03 07:06:22.083232 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.083199 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:06:22.127159 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.127136 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.145579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.145554 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:06:22.193996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.193968 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 07:06:22.193996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.193978 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t675q\"" Nov 03 07:06:22.217938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.217913 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbq2z\" (UniqueName: \"kubernetes.io/projected/91b42406-d50b-429d-b9e9-a1079b1dc162-kube-api-access-dbq2z\") pod \"limitador-limitador-7c9b87d574-lwnr4\" (UID: \"91b42406-d50b-429d-b9e9-a1079b1dc162\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.218100 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.217972 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91b42406-d50b-429d-b9e9-a1079b1dc162-config-file\") pod \"limitador-limitador-7c9b87d574-lwnr4\" (UID: \"91b42406-d50b-429d-b9e9-a1079b1dc162\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.318647 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.318609 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91b42406-d50b-429d-b9e9-a1079b1dc162-config-file\") pod \"limitador-limitador-7c9b87d574-lwnr4\" (UID: \"91b42406-d50b-429d-b9e9-a1079b1dc162\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.318898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.318867 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbq2z\" (UniqueName: \"kubernetes.io/projected/91b42406-d50b-429d-b9e9-a1079b1dc162-kube-api-access-dbq2z\") pod \"limitador-limitador-7c9b87d574-lwnr4\" (UID: \"91b42406-d50b-429d-b9e9-a1079b1dc162\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.319318 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.319300 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91b42406-d50b-429d-b9e9-a1079b1dc162-config-file\") pod \"limitador-limitador-7c9b87d574-lwnr4\" (UID: \"91b42406-d50b-429d-b9e9-a1079b1dc162\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.382945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.382878 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbq2z\" (UniqueName: \"kubernetes.io/projected/91b42406-d50b-429d-b9e9-a1079b1dc162-kube-api-access-dbq2z\") pod \"limitador-limitador-7c9b87d574-lwnr4\" (UID: \"91b42406-d50b-429d-b9e9-a1079b1dc162\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.389931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.389898 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:06:22.453047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.453001 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:22.843774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:22.843744 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:06:22.846877 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:06:22.846818 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b42406_d50b_429d_b9e9_a1079b1dc162.slice/crio-eccfb2746bbeb81339baa170f969a0f083bcb6bfbcfe04ba616385dc8864069a WatchSource:0}: Error finding container eccfb2746bbeb81339baa170f969a0f083bcb6bfbcfe04ba616385dc8864069a: Status 404 returned error can't find the container with id eccfb2746bbeb81339baa170f969a0f083bcb6bfbcfe04ba616385dc8864069a Nov 03 07:06:23.081924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:23.081881 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:23.332985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:23.332955 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" event={"ID":"91b42406-d50b-429d-b9e9-a1079b1dc162","Type":"ContainerStarted","Data":"eccfb2746bbeb81339baa170f969a0f083bcb6bfbcfe04ba616385dc8864069a"} Nov 03 07:06:23.334915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:23.334888 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:25.346023 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:25.345976 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" event={"ID":"91b42406-d50b-429d-b9e9-a1079b1dc162","Type":"ContainerStarted","Data":"a461087f6724c67189d65ec5e9da9b10c755994c7803066d07cc166189414a8f"} Nov 03 07:06:25.346410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:25.346135 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:25.469879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:25.469832 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" podStartSLOduration=1.778719147 podStartE2EDuration="3.469818913s" totalImagesPullingTime="1.691099766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:25.466755048 +0000 UTC m=+445.400283704" watchObservedRunningTime="2026-11-03 07:06:25.469818913 +0000 UTC m=+445.403347568" Nov 03 07:06:26.687166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:26.687134 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:26.785188 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:26.785159 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sfcmr"] Nov 03 07:06:26.901148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:26.901113 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:29.696303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:29.696206 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:29.891747 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:29.891703 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:31.378955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:31.378911 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" event={"ID":"5842ddf7-7844-4027-81e5-93fb95dd9abb","Type":"ContainerStarted","Data":"b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380"} Nov 03 07:06:31.378955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:31.378921 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" podUID="5842ddf7-7844-4027-81e5-93fb95dd9abb" containerName="authorino" containerID="cri-o://b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380" gracePeriod=30 Nov 03 07:06:31.498876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:31.498832 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" podStartSLOduration=1.8085204959999999 podStartE2EDuration="12.498818811s" totalImagesPullingTime="10.690298315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:31.497559417 +0000 UTC m=+451.431088074" watchObservedRunningTime="2026-11-03 07:06:31.498818811 +0000 UTC m=+451.432347467" Nov 03 07:06:31.631970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:31.631907 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" Nov 03 07:06:31.699677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:31.699655 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5842ddf7-7844-4027-81e5-93fb95dd9abb-kube-api-access-dst5b\" (UniqueName: \"kubernetes.io/projected/5842ddf7-7844-4027-81e5-93fb95dd9abb-kube-api-access-dst5b\") pod \"5842ddf7-7844-4027-81e5-93fb95dd9abb\" (UID: \"5842ddf7-7844-4027-81e5-93fb95dd9abb\") " Nov 03 07:06:31.701872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:31.701847 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5842ddf7-7844-4027-81e5-93fb95dd9abb-kube-api-access-dst5b" pod "5842ddf7-7844-4027-81e5-93fb95dd9abb" (UID: "5842ddf7-7844-4027-81e5-93fb95dd9abb"). InnerVolumeSpecName "kube-api-access-dst5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:06:31.801061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:31.800997 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dst5b\" (UniqueName: \"kubernetes.io/projected/5842ddf7-7844-4027-81e5-93fb95dd9abb-kube-api-access-dst5b\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:06:32.384900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.384865 4925 generic.go:356] "Generic (PLEG): container finished" podID="5842ddf7-7844-4027-81e5-93fb95dd9abb" containerID="b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380" exitCode=0 Nov 03 07:06:32.385393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.384923 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" Nov 03 07:06:32.385393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.384933 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" event={"ID":"5842ddf7-7844-4027-81e5-93fb95dd9abb","Type":"ContainerDied","Data":"b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380"} Nov 03 07:06:32.385393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.384965 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sfcmr" event={"ID":"5842ddf7-7844-4027-81e5-93fb95dd9abb","Type":"ContainerDied","Data":"94b37a4b74de645b7b870949adc733d978419c57b68cbbc9764a4506c7530831"} Nov 03 07:06:32.385393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.384983 4925 scope.go:122] "RemoveContainer" containerID="b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380" Nov 03 07:06:32.396238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.395340 4925 scope.go:122] "RemoveContainer" containerID="b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380" Nov 03 07:06:32.396238 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:06:32.396227 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380\": container with ID starting with b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380 not found: ID does not exist" containerID="b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380" Nov 03 07:06:32.396334 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.396252 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380"} err="failed to get container status \"b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380\": rpc error: code = NotFound desc = could not find container \"b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380\": container with ID starting with b433e7ec5b11d64e56e9fc94c7c1af37dce728f4adf7e6f1ca365ce8ad520380 not found: ID does not exist" Nov 03 07:06:32.554786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.554706 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sfcmr"] Nov 03 07:06:32.557692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:32.557668 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sfcmr"] Nov 03 07:06:34.191168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:34.189132 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5842ddf7-7844-4027-81e5-93fb95dd9abb" path="/var/lib/kubelet/pods/5842ddf7-7844-4027-81e5-93fb95dd9abb/volumes" Nov 03 07:06:35.325725 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:35.325689 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:35.543960 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:35.543904 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:36.352612 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:36.352581 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lwnr4" Nov 03 07:06:39.138071 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:06:39.138038 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:06:40.029191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:40.029154 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:40.270788 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:40.270748 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:40.839161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:40.839128 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:06:52.660708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:06:52.660665 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:01.922992 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:01.922945 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:02.118662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.118629 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:02.303197 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.303117 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:02.378255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.378223 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d848cbdf4-xpwnt"] Nov 03 07:07:02.378585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.378573 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5842ddf7-7844-4027-81e5-93fb95dd9abb" containerName="authorino" Nov 03 07:07:02.378627 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.378586 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5842ddf7-7844-4027-81e5-93fb95dd9abb" containerName="authorino" Nov 03 07:07:02.378666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.378656 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5842ddf7-7844-4027-81e5-93fb95dd9abb" containerName="authorino" Nov 03 07:07:02.382494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.382477 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.438563 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.438526 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 07:07:02.438563 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.438561 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 07:07:02.438764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.438596 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 07:07:02.438764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.438590 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 07:07:02.438764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.438625 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 07:07:02.438764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.438600 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7pc74\"" Nov 03 07:07:02.442831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.442809 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-xpwnt"] Nov 03 07:07:02.474574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.474547 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-metrics-certs\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.474680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.474621 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-webhook-cert\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.474754 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.474731 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwp5\" (UniqueName: \"kubernetes.io/projected/d393385c-b146-42fa-8963-21439be22464-kube-api-access-rmwp5\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.497726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.497695 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:02.575396 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.575373 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-webhook-cert\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.575539 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.575416 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwp5\" (UniqueName: \"kubernetes.io/projected/d393385c-b146-42fa-8963-21439be22464-kube-api-access-rmwp5\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.575539 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.575451 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-metrics-certs\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.578049 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.578029 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-webhook-cert\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.578149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.578067 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-metrics-certs\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.640195 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.640171 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwp5\" (UniqueName: \"kubernetes.io/projected/d393385c-b146-42fa-8963-21439be22464-kube-api-access-rmwp5\") pod \"maas-controller-d848cbdf4-xpwnt\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:02.695523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:02.695497 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:03.120649 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:03.120624 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-xpwnt"] Nov 03 07:07:03.122499 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:07:03.122474 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd393385c_b146_42fa_8963_21439be22464.slice/crio-d55c93a78d3b1155a8ad30647dddb5a18300c35097ee847446eeaaa6ced42b3a WatchSource:0}: Error finding container d55c93a78d3b1155a8ad30647dddb5a18300c35097ee847446eeaaa6ced42b3a: Status 404 returned error can't find the container with id d55c93a78d3b1155a8ad30647dddb5a18300c35097ee847446eeaaa6ced42b3a Nov 03 07:07:03.535070 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:03.534986 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" event={"ID":"d393385c-b146-42fa-8963-21439be22464","Type":"ContainerStarted","Data":"d55c93a78d3b1155a8ad30647dddb5a18300c35097ee847446eeaaa6ced42b3a"} Nov 03 07:07:08.195057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:08.195025 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:08.416757 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:08.416703 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:08.563152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:08.563125 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" event={"ID":"d393385c-b146-42fa-8963-21439be22464","Type":"ContainerStarted","Data":"913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc"} Nov 03 07:07:08.563329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:08.563274 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:08.689599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:08.689542 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" podStartSLOduration=2.061508834 podStartE2EDuration="6.689530018s" totalImagesPullingTime="4.628021184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:08.687266789 +0000 UTC m=+488.620795446" watchObservedRunningTime="2026-11-03 07:07:08.689530018 +0000 UTC m=+488.623058674" Nov 03 07:07:09.137114 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:07:09.137079 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:07:11.380374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:11.380297 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:11.619982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:11.619949 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:19.570270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:19.570242 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:07:30.024690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:30.024648 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:30.333699 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:30.333668 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:30.535394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:30.535366 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:30.730383 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:30.730283 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:30.932929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:30.932886 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:31.024188 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.024112 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp"] Nov 03 07:07:31.027725 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.027706 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.086840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.086812 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp"] Nov 03 07:07:31.111129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.111103 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a03a6544-fb55-423b-8ad2-2e32d1414628-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mm9hp\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.111261 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.111134 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstrb\" (UniqueName: \"kubernetes.io/projected/a03a6544-fb55-423b-8ad2-2e32d1414628-kube-api-access-jstrb\") pod \"llmisvc-controller-manager-869d5dcc5d-mm9hp\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.135261 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.135212 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 07:07:31.135386 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.135212 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-f2bhm\"" Nov 03 07:07:31.186936 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.186906 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:31.211651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.211619 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a03a6544-fb55-423b-8ad2-2e32d1414628-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mm9hp\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.211651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.211650 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstrb\" (UniqueName: \"kubernetes.io/projected/a03a6544-fb55-423b-8ad2-2e32d1414628-kube-api-access-jstrb\") pod \"llmisvc-controller-manager-869d5dcc5d-mm9hp\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.214339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.214305 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a03a6544-fb55-423b-8ad2-2e32d1414628-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mm9hp\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.278451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.278373 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstrb\" (UniqueName: \"kubernetes.io/projected/a03a6544-fb55-423b-8ad2-2e32d1414628-kube-api-access-jstrb\") pod \"llmisvc-controller-manager-869d5dcc5d-mm9hp\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.339979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.339952 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:31.719399 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:31.719369 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp"] Nov 03 07:07:31.721145 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:07:31.721119 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda03a6544_fb55_423b_8ad2_2e32d1414628.slice/crio-fefe1c194d9410ca9d98214b627af57877923392776413efce20f2183ea4581c WatchSource:0}: Error finding container fefe1c194d9410ca9d98214b627af57877923392776413efce20f2183ea4581c: Status 404 returned error can't find the container with id fefe1c194d9410ca9d98214b627af57877923392776413efce20f2183ea4581c Nov 03 07:07:32.397845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:32.397813 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:32.694323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:32.694231 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" event={"ID":"a03a6544-fb55-423b-8ad2-2e32d1414628","Type":"ContainerStarted","Data":"fefe1c194d9410ca9d98214b627af57877923392776413efce20f2183ea4581c"} Nov 03 07:07:34.706782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:34.706756 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" event={"ID":"a03a6544-fb55-423b-8ad2-2e32d1414628","Type":"ContainerStarted","Data":"6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf"} Nov 03 07:07:34.707129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:34.706892 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:07:34.830452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:34.830404 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" podStartSLOduration=1.947585211 podStartE2EDuration="4.830391029s" totalImagesPullingTime="2.882805818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:34.829994931 +0000 UTC m=+514.763523586" watchObservedRunningTime="2026-11-03 07:07:34.830391029 +0000 UTC m=+514.763919712" Nov 03 07:07:37.817526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:37.817470 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:38.082613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:38.082523 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:38.286339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:38.286307 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:38.501551 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:38.501449 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:38.700548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:38.700516 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:38.917731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:38.917692 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:39.111149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:39.111115 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:39.136517 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:07:39.136487 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:07:41.552447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:41.552410 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:41.759964 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:41.759931 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:43.720362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:43.720331 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:43.973625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:43.973547 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:44.176088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:44.176052 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:50.597993 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:50.597955 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:07:53.005297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:53.005262 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:57.660979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:57.660928 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:57.901975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:57.901934 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-x85fk"] Nov 03 07:07:57.905877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:57.905853 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" Nov 03 07:07:57.912355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:57.912297 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:07:57.929673 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:57.929649 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxl8d\" (UniqueName: \"kubernetes.io/projected/5b87e37c-682d-44be-a8d7-d43da21cf3c6-kube-api-access-qxl8d\") pod \"authorino-6f6944bcdd-x85fk\" (UID: \"5b87e37c-682d-44be-a8d7-d43da21cf3c6\") " pod="kuadrant-system/authorino-6f6944bcdd-x85fk" Nov 03 07:07:57.965645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:57.965610 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-x85fk"] Nov 03 07:07:58.014211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.014181 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zmffl\"" Nov 03 07:07:58.031143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.031116 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxl8d\" (UniqueName: \"kubernetes.io/projected/5b87e37c-682d-44be-a8d7-d43da21cf3c6-kube-api-access-qxl8d\") pod \"authorino-6f6944bcdd-x85fk\" (UID: \"5b87e37c-682d-44be-a8d7-d43da21cf3c6\") " pod="kuadrant-system/authorino-6f6944bcdd-x85fk" Nov 03 07:07:58.096183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.096146 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxl8d\" (UniqueName: \"kubernetes.io/projected/5b87e37c-682d-44be-a8d7-d43da21cf3c6-kube-api-access-qxl8d\") pod \"authorino-6f6944bcdd-x85fk\" (UID: \"5b87e37c-682d-44be-a8d7-d43da21cf3c6\") " pod="kuadrant-system/authorino-6f6944bcdd-x85fk" Nov 03 07:07:58.218998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.218925 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" Nov 03 07:07:58.282050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.281467 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-x85fk"] Nov 03 07:07:58.313581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.313550 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b9d9f76f-p5dr2"] Nov 03 07:07:58.317111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.317089 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" Nov 03 07:07:58.333911 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.333890 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrdh\" (UniqueName: \"kubernetes.io/projected/ed539012-bd1c-48c8-ab94-e72c3f5441b2-kube-api-access-qqrdh\") pod \"authorino-6b9d9f76f-p5dr2\" (UID: \"ed539012-bd1c-48c8-ab94-e72c3f5441b2\") " pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" Nov 03 07:07:58.375259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.375235 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b9d9f76f-p5dr2"] Nov 03 07:07:58.435099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.435071 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrdh\" (UniqueName: \"kubernetes.io/projected/ed539012-bd1c-48c8-ab94-e72c3f5441b2-kube-api-access-qqrdh\") pod \"authorino-6b9d9f76f-p5dr2\" (UID: \"ed539012-bd1c-48c8-ab94-e72c3f5441b2\") " pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" Nov 03 07:07:58.499662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.499596 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrdh\" (UniqueName: \"kubernetes.io/projected/ed539012-bd1c-48c8-ab94-e72c3f5441b2-kube-api-access-qqrdh\") pod \"authorino-6b9d9f76f-p5dr2\" (UID: \"ed539012-bd1c-48c8-ab94-e72c3f5441b2\") " pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" Nov 03 07:07:58.598405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.598361 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-x85fk"] Nov 03 07:07:58.600968 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:07:58.600937 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b87e37c_682d_44be_a8d7_d43da21cf3c6.slice/crio-d93082c3ec2aa40ed393f14269493b11fbd03de6ad9f0963c9c27a0575ecdab2 WatchSource:0}: Error finding container d93082c3ec2aa40ed393f14269493b11fbd03de6ad9f0963c9c27a0575ecdab2: Status 404 returned error can't find the container with id d93082c3ec2aa40ed393f14269493b11fbd03de6ad9f0963c9c27a0575ecdab2 Nov 03 07:07:58.631246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.631218 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" Nov 03 07:07:58.761938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.761851 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b9d9f76f-p5dr2"] Nov 03 07:07:58.799714 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.799683 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76d78cff-x92sw"] Nov 03 07:07:58.804196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.804171 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:58.832474 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.832441 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" event={"ID":"5b87e37c-682d-44be-a8d7-d43da21cf3c6","Type":"ContainerStarted","Data":"d93082c3ec2aa40ed393f14269493b11fbd03de6ad9f0963c9c27a0575ecdab2"} Nov 03 07:07:58.839814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.839792 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-tls-cert\") pod \"authorino-76d78cff-x92sw\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:58.839910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.839870 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5qrh\" (UniqueName: \"kubernetes.io/projected/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-kube-api-access-v5qrh\") pod \"authorino-76d78cff-x92sw\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:58.861287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.861260 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d78cff-x92sw"] Nov 03 07:07:58.862733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.862713 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 07:07:58.940547 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.940519 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5qrh\" (UniqueName: \"kubernetes.io/projected/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-kube-api-access-v5qrh\") pod \"authorino-76d78cff-x92sw\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:58.940699 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.940571 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-tls-cert\") pod \"authorino-76d78cff-x92sw\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:58.943307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:58.943282 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-tls-cert\") pod \"authorino-76d78cff-x92sw\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:59.009197 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.009166 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5qrh\" (UniqueName: \"kubernetes.io/projected/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-kube-api-access-v5qrh\") pod \"authorino-76d78cff-x92sw\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:59.018579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.018560 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b9d9f76f-p5dr2"] Nov 03 07:07:59.021112 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:07:59.021090 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded539012_bd1c_48c8_ab94_e72c3f5441b2.slice/crio-714f8c5d72dd08043c1aac501faec30e8913fbace6a172495a6d20075b8b13d2 WatchSource:0}: Error finding container 714f8c5d72dd08043c1aac501faec30e8913fbace6a172495a6d20075b8b13d2: Status 404 returned error can't find the container with id 714f8c5d72dd08043c1aac501faec30e8913fbace6a172495a6d20075b8b13d2 Nov 03 07:07:59.119616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.119590 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:07:59.499572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.498209 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d78cff-x92sw"] Nov 03 07:07:59.499987 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:07:59.499961 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf6f5e33_e75e_4fc3_9ceb_a34de721b8a3.slice/crio-a70028f2357932f6c8cfbca94cdb52ebfe7cb1bd03ec816213ac81d1f24e01f2 WatchSource:0}: Error finding container a70028f2357932f6c8cfbca94cdb52ebfe7cb1bd03ec816213ac81d1f24e01f2: Status 404 returned error can't find the container with id a70028f2357932f6c8cfbca94cdb52ebfe7cb1bd03ec816213ac81d1f24e01f2 Nov 03 07:07:59.839095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.839064 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d78cff-x92sw" event={"ID":"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3","Type":"ContainerStarted","Data":"a70028f2357932f6c8cfbca94cdb52ebfe7cb1bd03ec816213ac81d1f24e01f2"} Nov 03 07:07:59.840677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.840652 4925 generic.go:356] "Generic (PLEG): container finished" podID="da9e5265-25ca-4578-9533-096c5818509d" containerID="bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91" exitCode=0 Nov 03 07:07:59.840799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.840731 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" event={"ID":"da9e5265-25ca-4578-9533-096c5818509d","Type":"ContainerDied","Data":"bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91"} Nov 03 07:07:59.841163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.841138 4925 scope.go:122] "RemoveContainer" containerID="bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91" Nov 03 07:07:59.842093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.842070 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" event={"ID":"ed539012-bd1c-48c8-ab94-e72c3f5441b2","Type":"ContainerStarted","Data":"714f8c5d72dd08043c1aac501faec30e8913fbace6a172495a6d20075b8b13d2"} Nov 03 07:07:59.842253 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.842226 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:07:59.842320 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.842300 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:07:59.842363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:07:59.842331 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:08:00.878038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:00.876583 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" event={"ID":"ed539012-bd1c-48c8-ab94-e72c3f5441b2","Type":"ContainerStarted","Data":"874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a"} Nov 03 07:08:00.878038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:00.876775 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" podUID="ed539012-bd1c-48c8-ab94-e72c3f5441b2" containerName="authorino" containerID="cri-o://874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a" gracePeriod=30 Nov 03 07:08:00.882949 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:00.879749 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" event={"ID":"5b87e37c-682d-44be-a8d7-d43da21cf3c6","Type":"ContainerStarted","Data":"628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d"} Nov 03 07:08:00.882949 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:00.879887 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" podUID="5b87e37c-682d-44be-a8d7-d43da21cf3c6" containerName="authorino" containerID="cri-o://628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d" gracePeriod=30 Nov 03 07:08:00.890846 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:00.890820 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" event={"ID":"da9e5265-25ca-4578-9533-096c5818509d","Type":"ContainerStarted","Data":"e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714"} Nov 03 07:08:01.122449 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.122391 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" podStartSLOduration=1.976420508 podStartE2EDuration="3.12237315s" totalImagesPullingTime="1.145952642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:08:00.995643092 +0000 UTC m=+540.929171750" watchObservedRunningTime="2026-11-03 07:08:01.12237315 +0000 UTC m=+541.055901807" Nov 03 07:08:01.206659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.206639 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" Nov 03 07:08:01.209360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.209340 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" Nov 03 07:08:01.270071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.254926 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" podStartSLOduration=2.732907944 podStartE2EDuration="4.254905695s" totalImagesPullingTime="1.521997751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:08:01.123145362 +0000 UTC m=+541.056674017" watchObservedRunningTime="2026-11-03 07:08:01.254905695 +0000 UTC m=+541.188434351" Nov 03 07:08:01.278514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.278481 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b87e37c-682d-44be-a8d7-d43da21cf3c6-kube-api-access-qxl8d\" (UniqueName: \"kubernetes.io/projected/5b87e37c-682d-44be-a8d7-d43da21cf3c6-kube-api-access-qxl8d\") pod \"5b87e37c-682d-44be-a8d7-d43da21cf3c6\" (UID: \"5b87e37c-682d-44be-a8d7-d43da21cf3c6\") " Nov 03 07:08:01.278672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.278563 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed539012-bd1c-48c8-ab94-e72c3f5441b2-kube-api-access-qqrdh\" (UniqueName: \"kubernetes.io/projected/ed539012-bd1c-48c8-ab94-e72c3f5441b2-kube-api-access-qqrdh\") pod \"ed539012-bd1c-48c8-ab94-e72c3f5441b2\" (UID: \"ed539012-bd1c-48c8-ab94-e72c3f5441b2\") " Nov 03 07:08:01.280775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.280744 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b87e37c-682d-44be-a8d7-d43da21cf3c6-kube-api-access-qxl8d" pod "5b87e37c-682d-44be-a8d7-d43da21cf3c6" (UID: "5b87e37c-682d-44be-a8d7-d43da21cf3c6"). InnerVolumeSpecName "kube-api-access-qxl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:08:01.280883 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.280826 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed539012-bd1c-48c8-ab94-e72c3f5441b2-kube-api-access-qqrdh" pod "ed539012-bd1c-48c8-ab94-e72c3f5441b2" (UID: "ed539012-bd1c-48c8-ab94-e72c3f5441b2"). InnerVolumeSpecName "kube-api-access-qqrdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:08:01.380120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.380043 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxl8d\" (UniqueName: \"kubernetes.io/projected/5b87e37c-682d-44be-a8d7-d43da21cf3c6-kube-api-access-qxl8d\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:08:01.380120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.380070 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrdh\" (UniqueName: \"kubernetes.io/projected/ed539012-bd1c-48c8-ab94-e72c3f5441b2-kube-api-access-qqrdh\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:08:01.697860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.697766 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:08:01.698053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.697997 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 07:08:01.698140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.698074 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 07:08:01.898145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.898111 4925 generic.go:356] "Generic (PLEG): container finished" podID="ed539012-bd1c-48c8-ab94-e72c3f5441b2" containerID="874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a" exitCode=0 Nov 03 07:08:01.898593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.898172 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" Nov 03 07:08:01.898593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.898205 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" event={"ID":"ed539012-bd1c-48c8-ab94-e72c3f5441b2","Type":"ContainerDied","Data":"874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a"} Nov 03 07:08:01.898593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.898245 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b9d9f76f-p5dr2" event={"ID":"ed539012-bd1c-48c8-ab94-e72c3f5441b2","Type":"ContainerDied","Data":"714f8c5d72dd08043c1aac501faec30e8913fbace6a172495a6d20075b8b13d2"} Nov 03 07:08:01.898593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.898302 4925 scope.go:122] "RemoveContainer" containerID="874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a" Nov 03 07:08:01.899987 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.899966 4925 generic.go:356] "Generic (PLEG): container finished" podID="5b87e37c-682d-44be-a8d7-d43da21cf3c6" containerID="628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d" exitCode=0 Nov 03 07:08:01.900124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.900066 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" event={"ID":"5b87e37c-682d-44be-a8d7-d43da21cf3c6","Type":"ContainerDied","Data":"628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d"} Nov 03 07:08:01.900124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.900099 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" event={"ID":"5b87e37c-682d-44be-a8d7-d43da21cf3c6","Type":"ContainerDied","Data":"d93082c3ec2aa40ed393f14269493b11fbd03de6ad9f0963c9c27a0575ecdab2"} Nov 03 07:08:01.900124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.900071 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-x85fk" Nov 03 07:08:01.909399 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.909375 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d78cff-x92sw" event={"ID":"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3","Type":"ContainerStarted","Data":"be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8"} Nov 03 07:08:01.909611 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.909560 4925 scope.go:122] "RemoveContainer" containerID="874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a" Nov 03 07:08:01.909887 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:08:01.909857 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a\": container with ID starting with 874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a not found: ID does not exist" containerID="874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a" Nov 03 07:08:01.910002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.909897 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a"} err="failed to get container status \"874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a\": rpc error: code = NotFound desc = could not find container \"874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a\": container with ID starting with 874c4fe33aba8d83de454e6d55bd5f5b4f75749728474fa577e004d413ec786a not found: ID does not exist" Nov 03 07:08:01.910002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.909916 4925 scope.go:122] "RemoveContainer" containerID="628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d" Nov 03 07:08:01.919687 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.919668 4925 scope.go:122] "RemoveContainer" containerID="628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d" Nov 03 07:08:01.919985 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:08:01.919965 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d\": container with ID starting with 628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d not found: ID does not exist" containerID="628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d" Nov 03 07:08:01.920059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:01.919992 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d"} err="failed to get container status \"628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d\": rpc error: code = NotFound desc = could not find container \"628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d\": container with ID starting with 628be4a3a200971a8e2eb0bcab4e173ae57e62c55617ebe0c68611081c486c0d not found: ID does not exist" Nov 03 07:08:02.084419 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:02.084390 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-x85fk"] Nov 03 07:08:02.090089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:02.090061 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-x85fk"] Nov 03 07:08:02.193838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:02.193801 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b87e37c-682d-44be-a8d7-d43da21cf3c6" path="/var/lib/kubelet/pods/5b87e37c-682d-44be-a8d7-d43da21cf3c6/volumes" Nov 03 07:08:02.208442 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:02.208413 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b9d9f76f-p5dr2"] Nov 03 07:08:02.214857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:02.214835 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b9d9f76f-p5dr2"] Nov 03 07:08:02.696693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:02.696642 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 07:08:02.696856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:02.696714 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 07:08:03.696990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:03.696952 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 07:08:03.697414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:03.697046 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 07:08:04.191751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:04.191721 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed539012-bd1c-48c8-ab94-e72c3f5441b2" path="/var/lib/kubelet/pods/ed539012-bd1c-48c8-ab94-e72c3f5441b2/volumes" Nov 03 07:08:04.697189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:04.697150 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 07:08:04.697565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:04.697218 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 07:08:05.697073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:05.697037 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 07:08:05.697262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:05.697092 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 07:08:05.714262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:05.714237 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:08:05.838751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:05.838700 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-76d78cff-x92sw" podStartSLOduration=6.553605307 podStartE2EDuration="7.838687255s" totalImagesPullingTime="1.285081948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:08:02.336548913 +0000 UTC m=+542.270077568" watchObservedRunningTime="2026-11-03 07:08:05.838687255 +0000 UTC m=+545.772215910" Nov 03 07:08:06.700453 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:06.700425 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:08:06.700849 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:06.700708 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:08:06.701351 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:06.701337 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:08:09.138037 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:08:09.137981 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:08:12.021620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:12.021575 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:08:12.216731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:12.216683 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:08:39.138510 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:08:39.138478 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:08:43.757487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.757377 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9shxn"] Nov 03 07:08:43.758185 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.758165 4925 cpu_manager.go:414] "containerMap: removing container" podUID="ed539012-bd1c-48c8-ab94-e72c3f5441b2" containerName="authorino" Nov 03 07:08:43.758288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.758190 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed539012-bd1c-48c8-ab94-e72c3f5441b2" containerName="authorino" Nov 03 07:08:43.758288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.758224 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5b87e37c-682d-44be-a8d7-d43da21cf3c6" containerName="authorino" Nov 03 07:08:43.758288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.758231 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b87e37c-682d-44be-a8d7-d43da21cf3c6" containerName="authorino" Nov 03 07:08:43.758443 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.758321 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed539012-bd1c-48c8-ab94-e72c3f5441b2" containerName="authorino" Nov 03 07:08:43.758443 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.758337 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b87e37c-682d-44be-a8d7-d43da21cf3c6" containerName="authorino" Nov 03 07:08:43.761322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.761298 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:43.820540 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.820514 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9shxn"] Nov 03 07:08:43.940870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.940839 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7k65\" (UniqueName: \"kubernetes.io/projected/4da3a4da-2a87-4311-886a-7237c8e38e9e-kube-api-access-b7k65\") pod \"llmisvc-controller-manager-6466c554d8-9shxn\" (UID: \"4da3a4da-2a87-4311-886a-7237c8e38e9e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:43.941030 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:43.940949 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4da3a4da-2a87-4311-886a-7237c8e38e9e-cert\") pod \"llmisvc-controller-manager-6466c554d8-9shxn\" (UID: \"4da3a4da-2a87-4311-886a-7237c8e38e9e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:44.041948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:44.041874 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4da3a4da-2a87-4311-886a-7237c8e38e9e-cert\") pod \"llmisvc-controller-manager-6466c554d8-9shxn\" (UID: \"4da3a4da-2a87-4311-886a-7237c8e38e9e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:44.041948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:44.041921 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7k65\" (UniqueName: \"kubernetes.io/projected/4da3a4da-2a87-4311-886a-7237c8e38e9e-kube-api-access-b7k65\") pod \"llmisvc-controller-manager-6466c554d8-9shxn\" (UID: \"4da3a4da-2a87-4311-886a-7237c8e38e9e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:44.044494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:44.044468 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4da3a4da-2a87-4311-886a-7237c8e38e9e-cert\") pod \"llmisvc-controller-manager-6466c554d8-9shxn\" (UID: \"4da3a4da-2a87-4311-886a-7237c8e38e9e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:44.107432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:44.107407 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7k65\" (UniqueName: \"kubernetes.io/projected/4da3a4da-2a87-4311-886a-7237c8e38e9e-kube-api-access-b7k65\") pod \"llmisvc-controller-manager-6466c554d8-9shxn\" (UID: \"4da3a4da-2a87-4311-886a-7237c8e38e9e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:44.373900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:44.373870 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:44.748686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:44.748659 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9shxn"] Nov 03 07:08:44.750591 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:08:44.750566 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da3a4da_2a87_4311_886a_7237c8e38e9e.slice/crio-a1fccd05b1eb67ea227ec27c3757245798226f6cbecd32909d4560d35f85407d WatchSource:0}: Error finding container a1fccd05b1eb67ea227ec27c3757245798226f6cbecd32909d4560d35f85407d: Status 404 returned error can't find the container with id a1fccd05b1eb67ea227ec27c3757245798226f6cbecd32909d4560d35f85407d Nov 03 07:08:45.133262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:45.133227 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" event={"ID":"4da3a4da-2a87-4311-886a-7237c8e38e9e","Type":"ContainerStarted","Data":"a1fccd05b1eb67ea227ec27c3757245798226f6cbecd32909d4560d35f85407d"} Nov 03 07:08:46.141379 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:46.141346 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" event={"ID":"4da3a4da-2a87-4311-886a-7237c8e38e9e","Type":"ContainerStarted","Data":"1607e5220dff8419fa8dc4ae15a13d8864057e1779c15508c3cd0efb354ffd9d"} Nov 03 07:08:46.141791 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:46.141466 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:08:53.385125 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:53.385092 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:08:58.098210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:58.098169 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:08:59.399046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:08:59.398992 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:00.231103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:00.231070 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:09:00.232728 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:00.232693 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:09:00.234687 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:00.234667 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:09:00.236384 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:00.236361 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:09:05.897915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:05.897872 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:06.172000 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:06.171902 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:09.137584 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:09:09.137543 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:09:17.149053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.149022 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" Nov 03 07:09:17.276071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.276019 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9shxn" podStartSLOduration=33.501629221 podStartE2EDuration="34.275994002s" totalImagesPullingTime="774.364781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:08:46.262035545 +0000 UTC m=+586.195564192" watchObservedRunningTime="2026-11-03 07:09:17.275994002 +0000 UTC m=+617.209522677" Nov 03 07:09:17.327915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.327880 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp"] Nov 03 07:09:17.328115 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.328084 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" podUID="a03a6544-fb55-423b-8ad2-2e32d1414628" containerName="manager" containerID="cri-o://6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf" gracePeriod=30 Nov 03 07:09:17.582535 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.582511 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:09:17.612876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.612851 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03a6544-fb55-423b-8ad2-2e32d1414628-cert\" (UniqueName: \"kubernetes.io/secret/a03a6544-fb55-423b-8ad2-2e32d1414628-cert\") pod \"a03a6544-fb55-423b-8ad2-2e32d1414628\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " Nov 03 07:09:17.612970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.612962 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03a6544-fb55-423b-8ad2-2e32d1414628-kube-api-access-jstrb\" (UniqueName: \"kubernetes.io/projected/a03a6544-fb55-423b-8ad2-2e32d1414628-kube-api-access-jstrb\") pod \"a03a6544-fb55-423b-8ad2-2e32d1414628\" (UID: \"a03a6544-fb55-423b-8ad2-2e32d1414628\") " Nov 03 07:09:17.615212 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.615186 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03a6544-fb55-423b-8ad2-2e32d1414628-kube-api-access-jstrb" pod "a03a6544-fb55-423b-8ad2-2e32d1414628" (UID: "a03a6544-fb55-423b-8ad2-2e32d1414628"). InnerVolumeSpecName "kube-api-access-jstrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:09:17.615323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.615186 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03a6544-fb55-423b-8ad2-2e32d1414628-cert" pod "a03a6544-fb55-423b-8ad2-2e32d1414628" (UID: "a03a6544-fb55-423b-8ad2-2e32d1414628"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:09:17.713785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.713715 4925 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a03a6544-fb55-423b-8ad2-2e32d1414628-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:09:17.713785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:17.713740 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jstrb\" (UniqueName: \"kubernetes.io/projected/a03a6544-fb55-423b-8ad2-2e32d1414628-kube-api-access-jstrb\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:09:18.265032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.264973 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:18.313768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.313743 4925 generic.go:356] "Generic (PLEG): container finished" podID="a03a6544-fb55-423b-8ad2-2e32d1414628" containerID="6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf" exitCode=0 Nov 03 07:09:18.313935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.313785 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" Nov 03 07:09:18.313935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.313834 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" event={"ID":"a03a6544-fb55-423b-8ad2-2e32d1414628","Type":"ContainerDied","Data":"6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf"} Nov 03 07:09:18.313935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.313889 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp" event={"ID":"a03a6544-fb55-423b-8ad2-2e32d1414628","Type":"ContainerDied","Data":"fefe1c194d9410ca9d98214b627af57877923392776413efce20f2183ea4581c"} Nov 03 07:09:18.313935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.313916 4925 scope.go:122] "RemoveContainer" containerID="6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf" Nov 03 07:09:18.322974 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.322952 4925 scope.go:122] "RemoveContainer" containerID="6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf" Nov 03 07:09:18.323258 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:09:18.323239 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf\": container with ID starting with 6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf not found: ID does not exist" containerID="6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf" Nov 03 07:09:18.323314 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.323266 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf"} err="failed to get container status \"6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf\": rpc error: code = NotFound desc = could not find container \"6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf\": container with ID starting with 6c9796a26eb272c09cb0ac0f26ebdce601247f71b8649a5425b3e59bd4ab99cf not found: ID does not exist" Nov 03 07:09:18.495414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.495339 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp"] Nov 03 07:09:18.545730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.545697 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mm9hp"] Nov 03 07:09:18.550352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:18.550323 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:20.190361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:20.190332 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03a6544-fb55-423b-8ad2-2e32d1414628" path="/var/lib/kubelet/pods/a03a6544-fb55-423b-8ad2-2e32d1414628/volumes" Nov 03 07:09:34.808951 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:34.808894 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:35.010103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:35.010061 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:35.822376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:35.822335 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:39.137603 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:09:39.137560 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:09:39.385884 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:39.385845 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:09:40.207709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:40.207667 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:44.643032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:44.640820 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:47.680884 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:47.680854 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:48.114602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:48.114528 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:09:51.631168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:51.631131 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:53.732856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:53.732814 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:55.320528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:55.320488 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:09:58.826955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:09:58.826870 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:02.174587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:02.174548 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:10:02.201998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:02.201955 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:09.137951 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:10:09.137910 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:10:15.481775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:15.481683 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:10:18.121142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:18.121103 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:18.384692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:18.384609 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:20.853675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:20.853644 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d78cff-x92sw"] Nov 03 07:10:20.854268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:20.853960 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-76d78cff-x92sw" podUID="af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" containerName="authorino" containerID="cri-o://be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8" gracePeriod=30 Nov 03 07:10:21.118833 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.118778 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:10:21.170102 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.170075 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-kube-api-access-v5qrh\" (UniqueName: \"kubernetes.io/projected/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-kube-api-access-v5qrh\") pod \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " Nov 03 07:10:21.170236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.170193 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-tls-cert\" (UniqueName: \"kubernetes.io/secret/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-tls-cert\") pod \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\" (UID: \"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3\") " Nov 03 07:10:21.172370 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.172341 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-kube-api-access-v5qrh" pod "af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" (UID: "af6f5e33-e75e-4fc3-9ceb-a34de721b8a3"). InnerVolumeSpecName "kube-api-access-v5qrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:10:21.181406 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.181379 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-tls-cert" pod "af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" (UID: "af6f5e33-e75e-4fc3-9ceb-a34de721b8a3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:10:21.271397 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.271364 4925 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-tls-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:10:21.271397 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.271392 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5qrh\" (UniqueName: \"kubernetes.io/projected/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3-kube-api-access-v5qrh\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:10:21.671468 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.671437 4925 generic.go:356] "Generic (PLEG): container finished" podID="af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" containerID="be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8" exitCode=0 Nov 03 07:10:21.671648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.671488 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d78cff-x92sw" Nov 03 07:10:21.671648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.671533 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d78cff-x92sw" event={"ID":"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3","Type":"ContainerDied","Data":"be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8"} Nov 03 07:10:21.671648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.671574 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d78cff-x92sw" event={"ID":"af6f5e33-e75e-4fc3-9ceb-a34de721b8a3","Type":"ContainerDied","Data":"a70028f2357932f6c8cfbca94cdb52ebfe7cb1bd03ec816213ac81d1f24e01f2"} Nov 03 07:10:21.671648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.671591 4925 scope.go:122] "RemoveContainer" containerID="be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8" Nov 03 07:10:21.680763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.680748 4925 scope.go:122] "RemoveContainer" containerID="be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8" Nov 03 07:10:21.680999 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:10:21.680977 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8\": container with ID starting with be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8 not found: ID does not exist" containerID="be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8" Nov 03 07:10:21.681093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.681041 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8"} err="failed to get container status \"be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8\": rpc error: code = NotFound desc = could not find container \"be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8\": container with ID starting with be17cb7c49393c5e935e8bed9a8a024f3c66b1020a6a0746ce4a342fb3fcb5b8 not found: ID does not exist" Nov 03 07:10:21.842866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.842834 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d78cff-x92sw"] Nov 03 07:10:21.893835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:21.893807 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76d78cff-x92sw"] Nov 03 07:10:22.189586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:22.189540 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" path="/var/lib/kubelet/pods/af6f5e33-e75e-4fc3-9ceb-a34de721b8a3/volumes" Nov 03 07:10:27.862102 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:27.862060 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:28.052918 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:28.052875 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:39.137328 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:10:39.137292 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:10:51.495694 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:51.495660 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:51.998831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:51.998793 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:52.194325 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:52.194283 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:53.580181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:53.580134 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:53.776957 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:53.776911 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:54.074480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:54.074450 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:55.201646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:55.201615 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:10:55.394038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:10:55.393974 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:09.138414 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:09.138376 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:11:18.985482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:18.985442 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:19.179880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:19.179842 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:19.652940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:19.652893 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:19.905363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:19.905270 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:20.091901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:20.091860 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:20.292405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:20.292321 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:20.476738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:20.476708 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:20.666275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:20.666237 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:20.857243 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:20.857201 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:21.064686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:21.064655 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:21.267620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:21.267588 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:21.457781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:21.457692 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:21.654408 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:21.654373 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:21.844743 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:21.844712 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:22.302636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:22.302566 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:24.470360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:24.470315 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:24.707119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:24.707089 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:25.123916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:25.123877 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:25.373686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:25.373635 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:25.957357 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:25.957314 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:26.202970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:26.202934 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:26.598412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:26.598370 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:26.788374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:26.788342 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:27.003946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:27.003859 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:28.882807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:28.882767 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:29.081231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:29.081178 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:29.478124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:29.478096 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:29.663790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:29.663749 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:29.856241 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:29.856212 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:30.785694 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:30.785660 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:30.971734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:30.971693 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:31.167537 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:31.167502 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:33.876702 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:33.876664 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:34.065605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:34.065566 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:36.278185 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:36.278150 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:36.524842 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:36.524799 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:36.740050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:36.739985 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:37.088845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:37.088789 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:37.297382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:37.297347 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:39.138722 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:39.138689 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:11:39.224196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:39.224162 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:11:41.448274 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.448176 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:41.680752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.680176 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv"] Nov 03 07:11:41.680752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.680644 4925 cpu_manager.go:414] "containerMap: removing container" podUID="af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" containerName="authorino" Nov 03 07:11:41.680752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.680660 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" containerName="authorino" Nov 03 07:11:41.680752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.680702 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a03a6544-fb55-423b-8ad2-2e32d1414628" containerName="manager" Nov 03 07:11:41.680752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.680708 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03a6544-fb55-423b-8ad2-2e32d1414628" containerName="manager" Nov 03 07:11:41.681327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.680777 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03a6544-fb55-423b-8ad2-2e32d1414628" containerName="manager" Nov 03 07:11:41.681327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.680788 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6f5e33-e75e-4fc3-9ceb-a34de721b8a3" containerName="authorino" Nov 03 07:11:41.684281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.684260 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.703179 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.703117 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:41.758457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.758426 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-metrics-certs\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.758599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.758465 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a3483b-f650-46e3-a927-002cd5cf94ca-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.758599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.758563 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcm5b\" (UniqueName: \"kubernetes.io/projected/e2a3483b-f650-46e3-a927-002cd5cf94ca-kube-api-access-hcm5b\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.758599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.758587 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.791845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.791813 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-1f2e7e-serving-cert\"" Nov 03 07:11:41.792045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.791904 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6zz26\"" Nov 03 07:11:41.792045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.791922 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 07:11:41.792045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.791936 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 07:11:41.792045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.791955 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 07:11:41.794023 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.793983 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv"] Nov 03 07:11:41.859056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.858996 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcm5b\" (UniqueName: \"kubernetes.io/projected/e2a3483b-f650-46e3-a927-002cd5cf94ca-kube-api-access-hcm5b\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.859250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.859066 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.859250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.859117 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-metrics-certs\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.859250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.859140 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a3483b-f650-46e3-a927-002cd5cf94ca-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.859250 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:41.859237 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-serving-cert: secret "maas-api-e2e-worker-w3-1f2e7e-serving-cert" not found Nov 03 07:11:41.859486 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:41.859326 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls podName:e2a3483b-f650-46e3-a927-002cd5cf94ca nodeName:}" failed. No retries permitted until 2026-11-03 07:11:42.359303869 +0000 UTC m=+762.292832520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls") pod "maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" (UID: "e2a3483b-f650-46e3-a927-002cd5cf94ca") : secret "maas-api-e2e-worker-w3-1f2e7e-serving-cert" not found Nov 03 07:11:41.859831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.859814 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a3483b-f650-46e3-a927-002cd5cf94ca-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.861738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.861713 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-metrics-certs\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.923713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.923686 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcm5b\" (UniqueName: \"kubernetes.io/projected/e2a3483b-f650-46e3-a927-002cd5cf94ca-kube-api-access-hcm5b\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:41.928404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:41.928372 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:42.128072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:42.128035 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:42.326229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:42.326194 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:42.363396 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:42.363368 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:42.365907 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:42.365879 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:42.599416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:42.599378 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:42.977272 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:42.977244 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv"] Nov 03 07:11:42.979501 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:11:42.979470 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a3483b_f650_46e3_a927_002cd5cf94ca.slice/crio-6d83e0d1e648e27fe6b1d66fbae826552b88a44ad71cf15c9d5be98f8d8c3de6 WatchSource:0}: Error finding container 6d83e0d1e648e27fe6b1d66fbae826552b88a44ad71cf15c9d5be98f8d8c3de6: Status 404 returned error can't find the container with id 6d83e0d1e648e27fe6b1d66fbae826552b88a44ad71cf15c9d5be98f8d8c3de6 Nov 03 07:11:42.980808 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:42.980786 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:11:43.113296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:43.113256 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" event={"ID":"e2a3483b-f650-46e3-a927-002cd5cf94ca","Type":"ContainerStarted","Data":"6d83e0d1e648e27fe6b1d66fbae826552b88a44ad71cf15c9d5be98f8d8c3de6"} Nov 03 07:11:45.127799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:45.127705 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" event={"ID":"e2a3483b-f650-46e3-a927-002cd5cf94ca","Type":"ContainerStarted","Data":"377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67"} Nov 03 07:11:45.128221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:45.127851 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:45.255601 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:45.255548 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" podStartSLOduration=2.397857883 podStartE2EDuration="4.255535545s" totalImagesPullingTime="1.857677662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:45.255288059 +0000 UTC m=+765.188816728" watchObservedRunningTime="2026-11-03 07:11:45.255535545 +0000 UTC m=+765.189064248" Nov 03 07:11:46.436899 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:46.436861 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:46.685321 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:46.685286 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:47.139857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.139827 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl"] Nov 03 07:11:47.144308 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.144284 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.199901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.199879 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6606d6-serving-cert\"" Nov 03 07:11:47.205193 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.205169 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl"] Nov 03 07:11:47.209111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.209084 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c123d0b-147b-4710-a0fc-0022ded11038-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.209246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.209223 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dx7t\" (UniqueName: \"kubernetes.io/projected/4c123d0b-147b-4710-a0fc-0022ded11038-kube-api-access-9dx7t\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.209311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.209258 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.209311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.209280 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-metrics-certs\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.309966 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.309933 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c123d0b-147b-4710-a0fc-0022ded11038-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.310170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.310025 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dx7t\" (UniqueName: \"kubernetes.io/projected/4c123d0b-147b-4710-a0fc-0022ded11038-kube-api-access-9dx7t\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.310170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.310054 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.310170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.310084 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-metrics-certs\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.310406 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:47.310386 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-serving-cert: secret "maas-api-e2e-worker-w0-6606d6-serving-cert" not found Nov 03 07:11:47.310472 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:47.310444 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls podName:4c123d0b-147b-4710-a0fc-0022ded11038 nodeName:}" failed. No retries permitted until 2026-11-03 07:11:47.810428814 +0000 UTC m=+767.743957449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls") pod "maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" (UID: "4c123d0b-147b-4710-a0fc-0022ded11038") : secret "maas-api-e2e-worker-w0-6606d6-serving-cert" not found Nov 03 07:11:47.311593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.311569 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c123d0b-147b-4710-a0fc-0022ded11038-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.313963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.313940 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-metrics-certs\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.374130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.374103 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dx7t\" (UniqueName: \"kubernetes.io/projected/4c123d0b-147b-4710-a0fc-0022ded11038-kube-api-access-9dx7t\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.424495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.424412 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:47.616958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.616921 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:47.815099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.815073 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.817598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.817578 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:47.983619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:47.983578 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:48.057389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:48.057351 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:48.312749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:48.312711 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:11:48.490970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:48.490943 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl"] Nov 03 07:11:48.492430 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:11:48.492403 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c123d0b_147b_4710_a0fc_0022ded11038.slice/crio-e2d069b4929e3076615850e99ba91735d6cdc37648209c39095ec6c475fa67d6 WatchSource:0}: Error finding container e2d069b4929e3076615850e99ba91735d6cdc37648209c39095ec6c475fa67d6: Status 404 returned error can't find the container with id e2d069b4929e3076615850e99ba91735d6cdc37648209c39095ec6c475fa67d6 Nov 03 07:11:49.132625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:49.132585 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:49.152690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:49.152650 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" event={"ID":"4c123d0b-147b-4710-a0fc-0022ded11038","Type":"ContainerStarted","Data":"e2d069b4929e3076615850e99ba91735d6cdc37648209c39095ec6c475fa67d6"} Nov 03 07:11:49.337395 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:49.337346 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:50.160513 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:50.160449 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" event={"ID":"4c123d0b-147b-4710-a0fc-0022ded11038","Type":"ContainerStarted","Data":"931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8"} Nov 03 07:11:50.160980 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:50.160533 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:50.289383 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:50.289320 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" podStartSLOduration=2.484901505 podStartE2EDuration="3.289305819s" totalImagesPullingTime="804.404314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:50.287253078 +0000 UTC m=+770.220781737" watchObservedRunningTime="2026-11-03 07:11:50.289305819 +0000 UTC m=+770.222834474" Nov 03 07:11:51.139727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:51.139702 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:11:51.846693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:51.846654 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:52.037779 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:52.037743 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:52.800164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:52.800129 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:53.002999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:53.002959 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:53.204258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:53.204221 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:54.422857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:54.422822 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:56.172044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:56.171991 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:11:56.919103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:56.919074 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:11:57.740515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.740483 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl"] Nov 03 07:11:57.744592 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.744571 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.803512 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.803488 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-3b41b9-serving-cert\"" Nov 03 07:11:57.806687 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.806662 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-metrics-certs\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.806830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.806696 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.806830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.806718 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmtw\" (UniqueName: \"kubernetes.io/projected/b61eccd2-6a03-4dd9-829f-414965b61fa1-kube-api-access-ffmtw\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.806830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.806791 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61eccd2-6a03-4dd9-829f-414965b61fa1-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.856692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.856656 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl"] Nov 03 07:11:57.908304 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.908275 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-metrics-certs\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.908304 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.908310 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.908508 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.908326 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmtw\" (UniqueName: \"kubernetes.io/projected/b61eccd2-6a03-4dd9-829f-414965b61fa1-kube-api-access-ffmtw\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.908508 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:57.908425 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-serving-cert: secret "maas-api-e2e-worker-w6-3b41b9-serving-cert" not found Nov 03 07:11:57.908508 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:11:57.908487 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls podName:b61eccd2-6a03-4dd9-829f-414965b61fa1 nodeName:}" failed. No retries permitted until 2026-11-03 07:11:58.408470747 +0000 UTC m=+778.341999394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls") pod "maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" (UID: "b61eccd2-6a03-4dd9-829f-414965b61fa1") : secret "maas-api-e2e-worker-w6-3b41b9-serving-cert" not found Nov 03 07:11:57.908656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.908531 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61eccd2-6a03-4dd9-829f-414965b61fa1-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.909202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.909175 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61eccd2-6a03-4dd9-829f-414965b61fa1-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.910867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.910843 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-metrics-certs\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:57.974783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:57.974752 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmtw\" (UniqueName: \"kubernetes.io/projected/b61eccd2-6a03-4dd9-829f-414965b61fa1-kube-api-access-ffmtw\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:58.013578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:58.013512 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:58.253678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:58.253639 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:11:58.414432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:58.414395 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:58.416887 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:58.416859 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:58.657819 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:58.657771 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:11:59.038550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:59.038523 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl"] Nov 03 07:11:59.039608 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:11:59.039580 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61eccd2_6a03_4dd9_829f_414965b61fa1.slice/crio-094e78d306eed043e2eb2012d85995feaec1760ecf9c84cda8019817ccdafbd4 WatchSource:0}: Error finding container 094e78d306eed043e2eb2012d85995feaec1760ecf9c84cda8019817ccdafbd4: Status 404 returned error can't find the container with id 094e78d306eed043e2eb2012d85995feaec1760ecf9c84cda8019817ccdafbd4 Nov 03 07:11:59.214370 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:11:59.214339 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" event={"ID":"b61eccd2-6a03-4dd9-829f-414965b61fa1","Type":"ContainerStarted","Data":"094e78d306eed043e2eb2012d85995feaec1760ecf9c84cda8019817ccdafbd4"} Nov 03 07:12:00.227907 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:00.227808 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" event={"ID":"b61eccd2-6a03-4dd9-829f-414965b61fa1","Type":"ContainerStarted","Data":"7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c"} Nov 03 07:12:00.228348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:00.227944 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:12:00.352563 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:00.352501 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" podStartSLOduration=2.555448181 podStartE2EDuration="3.352484419s" totalImagesPullingTime="797.036238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:12:00.349859418 +0000 UTC m=+780.283388073" watchObservedRunningTime="2026-11-03 07:12:00.352484419 +0000 UTC m=+780.286013076" Nov 03 07:12:03.041999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.041966 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd"] Nov 03 07:12:03.045707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.045686 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.101715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.101686 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-9697f7-serving-cert\"" Nov 03 07:12:03.106715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.106691 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd"] Nov 03 07:12:03.158913 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.158884 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-metrics-certs\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.159116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.158921 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdtgq\" (UniqueName: \"kubernetes.io/projected/8145ea1e-95c7-47d8-a304-473757d53b74-kube-api-access-vdtgq\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.159116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.158957 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.159251 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.159168 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8145ea1e-95c7-47d8-a304-473757d53b74-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.260391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.260353 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-metrics-certs\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.260391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.260399 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdtgq\" (UniqueName: \"kubernetes.io/projected/8145ea1e-95c7-47d8-a304-473757d53b74-kube-api-access-vdtgq\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.260645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.260448 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.260645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.260481 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8145ea1e-95c7-47d8-a304-473757d53b74-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.260645 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:12:03.260584 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-serving-cert: secret "maas-api-e2e-worker-w1-9697f7-serving-cert" not found Nov 03 07:12:03.260824 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:12:03.260672 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls podName:8145ea1e-95c7-47d8-a304-473757d53b74 nodeName:}" failed. No retries permitted until 2026-11-03 07:12:03.760650486 +0000 UTC m=+783.694179141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls") pod "maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" (UID: "8145ea1e-95c7-47d8-a304-473757d53b74") : secret "maas-api-e2e-worker-w1-9697f7-serving-cert" not found Nov 03 07:12:03.261239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.261217 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8145ea1e-95c7-47d8-a304-473757d53b74-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.263150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.263129 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-metrics-certs\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.289424 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.289393 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:03.325259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.325218 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdtgq\" (UniqueName: \"kubernetes.io/projected/8145ea1e-95c7-47d8-a304-473757d53b74-kube-api-access-vdtgq\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.486108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.486072 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:03.716530 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.716440 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:03.766741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.766708 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.769312 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.769291 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:03.959809 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:03.959771 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:04.346820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:04.346794 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd"] Nov 03 07:12:04.348870 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:12:04.348833 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8145ea1e_95c7_47d8_a304_473757d53b74.slice/crio-3dfc96bef6dbb441fe3f444cdd4c7b2dcaa21459200b229a6ab1c737b2438167 WatchSource:0}: Error finding container 3dfc96bef6dbb441fe3f444cdd4c7b2dcaa21459200b229a6ab1c737b2438167: Status 404 returned error can't find the container with id 3dfc96bef6dbb441fe3f444cdd4c7b2dcaa21459200b229a6ab1c737b2438167 Nov 03 07:12:05.257317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:05.257270 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" event={"ID":"8145ea1e-95c7-47d8-a304-473757d53b74","Type":"ContainerStarted","Data":"6ee1a4db8237800154a5d10e3014d99559fa1da9d34772977c1038ee87c615d1"} Nov 03 07:12:05.257503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:05.257428 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:05.257503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:05.257454 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" event={"ID":"8145ea1e-95c7-47d8-a304-473757d53b74","Type":"ContainerStarted","Data":"3dfc96bef6dbb441fe3f444cdd4c7b2dcaa21459200b229a6ab1c737b2438167"} Nov 03 07:12:05.381646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:05.381596 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" podStartSLOduration=2.5965286880000003 podStartE2EDuration="3.381581329s" totalImagesPullingTime="785.052641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:12:05.379596358 +0000 UTC m=+785.313125039" watchObservedRunningTime="2026-11-03 07:12:05.381581329 +0000 UTC m=+785.315109984" Nov 03 07:12:06.238134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:06.238108 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:12:08.550053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:08.549997 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:09.137490 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:12:09.137451 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:12:11.267773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:11.267742 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:12:13.629990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:13.629946 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:16.299919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:16.299885 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:12:18.726485 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:18.726439 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:19.960505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:19.960454 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:20.557985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:20.557944 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:20.765470 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:20.765436 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:24.118079 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.118043 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w"] Nov 03 07:12:24.118963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.118937 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:24.121880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.121865 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.155653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.155616 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-model-cache\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.155785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.155661 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-tmp-dir\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.155785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.155710 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-dshm\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.155785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.155753 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc85j\" (UniqueName: \"kubernetes.io/projected/28910ce8-0366-4962-94a8-4ed5667820a3-kube-api-access-jc85j\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.155950 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.155874 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-home\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.155950 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.155919 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28910ce8-0366-4962-94a8-4ed5667820a3-tls-certs\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.176988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.176967 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"kube-root-ca.crt\"" Nov 03 07:12:24.227802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.227773 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"default-dockercfg-td9rn\"" Nov 03 07:12:24.227802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.227787 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"openshift-service-ca.crt\"" Nov 03 07:12:24.230266 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.230245 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w"] Nov 03 07:12:24.256505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.256469 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-home\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.256505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.256511 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28910ce8-0366-4962-94a8-4ed5667820a3-tls-certs\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.256759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.256596 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-model-cache\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.256759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.256624 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-tmp-dir\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.256759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.256655 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-dshm\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.256759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.256685 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc85j\" (UniqueName: \"kubernetes.io/projected/28910ce8-0366-4962-94a8-4ed5667820a3-kube-api-access-jc85j\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.257147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.256977 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-home\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.257380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.257358 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-tmp-dir\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.257529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.257506 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-model-cache\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.259652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.259626 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-dshm\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.278764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.278731 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"simulator-w2-669d6b-kserve-self-signed-certs\"" Nov 03 07:12:24.289663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.289636 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28910ce8-0366-4962-94a8-4ed5667820a3-tls-certs\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.310909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.310882 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:24.323861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.323828 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc85j\" (UniqueName: \"kubernetes.io/projected/28910ce8-0366-4962-94a8-4ed5667820a3-kube-api-access-jc85j\") pod \"simulator-w2-669d6b-kserve-96bf6b7fd-whb5w\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.434177 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.434089 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:24.816739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:24.816715 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w"] Nov 03 07:12:24.818568 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:12:24.818545 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28910ce8_0366_4962_94a8_4ed5667820a3.slice/crio-a245b87258a28498fbecfd735df7f460de29c7ebf7702bdec9b91069b45b9a13 WatchSource:0}: Error finding container a245b87258a28498fbecfd735df7f460de29c7ebf7702bdec9b91069b45b9a13: Status 404 returned error can't find the container with id a245b87258a28498fbecfd735df7f460de29c7ebf7702bdec9b91069b45b9a13 Nov 03 07:12:25.384192 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:25.384159 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" event={"ID":"28910ce8-0366-4962-94a8-4ed5667820a3","Type":"ContainerStarted","Data":"a245b87258a28498fbecfd735df7f460de29c7ebf7702bdec9b91069b45b9a13"} Nov 03 07:12:26.797360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:26.797306 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:27.398200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:27.398107 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" event={"ID":"28910ce8-0366-4962-94a8-4ed5667820a3","Type":"ContainerStarted","Data":"35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae"} Nov 03 07:12:27.434478 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:27.434443 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:27.446722 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:27.446697 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:27.523240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:27.523188 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podStartSLOduration=1.212930315 podStartE2EDuration="3.523174544s" totalImagesPullingTime="2.310244229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:12:27.521972035 +0000 UTC m=+807.455500691" watchObservedRunningTime="2026-11-03 07:12:27.523174544 +0000 UTC m=+807.456703200" Nov 03 07:12:27.652002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:27.651909 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:29.775631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:29.775596 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:30.399186 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:30.399146 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:12:31.417695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:31.417640 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:32.051912 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:32.051875 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:32.297419 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:32.297385 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:34.434819 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:34.434786 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:34.448213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:34.448192 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:35.318089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:35.318051 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:35.446606 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:35.446572 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:12:36.503813 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:36.503781 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:37.145514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:37.145473 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:37.355237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:37.355197 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:39.137231 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:12:39.137183 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:12:41.150901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:41.150865 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:44.972115 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:44.972073 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:45.536936 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:45.536894 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:45.789967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:45.789877 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:46.715996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:46.715952 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:50.113829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:50.113792 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:50.801001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:50.800941 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:51.066356 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:51.066312 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:52.691504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:52.691471 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:54.440926 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.440891 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6"] Nov 03 07:12:54.445107 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.445081 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.505028 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.504985 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6"] Nov 03 07:12:54.527120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.527091 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tmp-dir\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.527252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.527130 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-model-cache\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.527293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.527272 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkf5w\" (UniqueName: \"kubernetes.io/projected/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-kube-api-access-xkf5w\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.527379 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.527303 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tls-certs\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.527379 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.527325 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-dshm\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.527456 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.527414 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-home\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.553593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.553562 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"simulator-w4-ffee69-kserve-self-signed-certs\"" Nov 03 07:12:54.553826 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.553599 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"kube-root-ca.crt\"" Nov 03 07:12:54.554203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.554173 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"openshift-service-ca.crt\"" Nov 03 07:12:54.554340 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.554254 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"default-dockercfg-r7hhd\"" Nov 03 07:12:54.627945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.627917 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkf5w\" (UniqueName: \"kubernetes.io/projected/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-kube-api-access-xkf5w\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.627945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.627952 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tls-certs\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.628234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.627971 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-dshm\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.628303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.628224 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-home\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.628303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.628286 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tmp-dir\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.628410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.628323 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-model-cache\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.628925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.628751 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-model-cache\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.628925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.628916 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tmp-dir\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.629169 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.629145 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-home\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.630636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.630604 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-dshm\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.630987 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.630964 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tls-certs\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.664069 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.664041 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:54.697691 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.697632 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkf5w\" (UniqueName: \"kubernetes.io/projected/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-kube-api-access-xkf5w\") pod \"simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.760587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.760546 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:54.889793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:54.889757 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:55.150249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:55.150214 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6"] Nov 03 07:12:55.153463 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:12:55.153437 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf80aa4db_3d6d_4f15_ab7c_09587d3285b4.slice/crio-1d1e60da13a1b67ced439a883be3b08b89559cd3fa000362fcafd421656ee623 WatchSource:0}: Error finding container 1d1e60da13a1b67ced439a883be3b08b89559cd3fa000362fcafd421656ee623: Status 404 returned error can't find the container with id 1d1e60da13a1b67ced439a883be3b08b89559cd3fa000362fcafd421656ee623 Nov 03 07:12:55.552840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:55.552727 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" event={"ID":"f80aa4db-3d6d-4f15-ab7c-09587d3285b4","Type":"ContainerStarted","Data":"5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c"} Nov 03 07:12:55.552840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:55.552782 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" event={"ID":"f80aa4db-3d6d-4f15-ab7c-09587d3285b4","Type":"ContainerStarted","Data":"1d1e60da13a1b67ced439a883be3b08b89559cd3fa000362fcafd421656ee623"} Nov 03 07:12:55.684819 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:55.684759 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podStartSLOduration=1.68474415 podStartE2EDuration="1.68474415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:12:55.683449101 +0000 UTC m=+835.616977758" watchObservedRunningTime="2026-11-03 07:12:55.68474415 +0000 UTC m=+835.618272807" Nov 03 07:12:55.761029 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:55.760987 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:12:56.028157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:56.028125 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:56.639278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:56.639240 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:56.859631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:56.859597 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:58.445845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:58.445807 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:59.328342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.328312 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7"] Nov 03 07:12:59.331963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.331946 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.394568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.394538 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"default-dockercfg-tkq4z\"" Nov 03 07:12:59.445341 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.445306 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"kube-root-ca.crt\"" Nov 03 07:12:59.445341 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.445312 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"simulator-w3-1f2e7e-kserve-self-signed-certs\"" Nov 03 07:12:59.445572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.445366 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"openshift-service-ca.crt\"" Nov 03 07:12:59.449464 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.449415 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7"] Nov 03 07:12:59.476431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.476401 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-home\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.476621 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.476552 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-model-cache\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.476783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.476720 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tmp-dir\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.476849 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.476833 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wz9\" (UniqueName: \"kubernetes.io/projected/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-kube-api-access-68wz9\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.476913 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.476865 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tls-certs\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.476913 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.476890 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-dshm\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.524352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.524238 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:12:59.577670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.577639 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wz9\" (UniqueName: \"kubernetes.io/projected/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-kube-api-access-68wz9\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.577846 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.577678 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tls-certs\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.577846 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.577697 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-dshm\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.577846 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.577757 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-home\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.578063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.577861 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-model-cache\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.578063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.577881 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tmp-dir\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.578457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.578376 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-model-cache\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.578581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.578483 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-home\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.578737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.578665 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tmp-dir\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.580376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.580347 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-dshm\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.580473 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.580425 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tls-certs\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.642643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.642614 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wz9\" (UniqueName: \"kubernetes.io/projected/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-kube-api-access-68wz9\") pod \"simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:12:59.644491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:12:59.644468 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:13:00.037059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:00.037032 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7"] Nov 03 07:13:00.038817 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:13:00.038789 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bc1842f_4ade_4a0b_9bc0_af7fabc0c92e.slice/crio-9691f6ddf90c0af2077b44683a7707a2adf0a4bf1927397580a13c975f37fdd3 WatchSource:0}: Error finding container 9691f6ddf90c0af2077b44683a7707a2adf0a4bf1927397580a13c975f37fdd3: Status 404 returned error can't find the container with id 9691f6ddf90c0af2077b44683a7707a2adf0a4bf1927397580a13c975f37fdd3 Nov 03 07:13:00.506200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:00.506169 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:13:00.588981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:00.588949 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" event={"ID":"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e","Type":"ContainerStarted","Data":"566230d1c5f8de8e78727b69cd44df0b852a158610fa7a901e38abdd3ec33e30"} Nov 03 07:13:00.588981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:00.588990 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" event={"ID":"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e","Type":"ContainerStarted","Data":"9691f6ddf90c0af2077b44683a7707a2adf0a4bf1927397580a13c975f37fdd3"} Nov 03 07:13:00.644742 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:00.644713 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:13:00.714227 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:00.714177 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podStartSLOduration=1.71416454 podStartE2EDuration="1.71416454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:13:00.712376542 +0000 UTC m=+840.645905197" watchObservedRunningTime="2026-11-03 07:13:00.71416454 +0000 UTC m=+840.647693196" Nov 03 07:13:03.759419 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:03.759380 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:04.348765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:04.348726 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:04.761699 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:04.761610 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:13:04.773954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:04.773933 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:13:05.626781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:05.626754 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:13:07.183808 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.183775 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s"] Nov 03 07:13:07.189229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.189206 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.248285 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.248250 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s"] Nov 03 07:13:07.295456 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.295426 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"default-dockercfg-wk9lj\"" Nov 03 07:13:07.295577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.295519 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"simulator-w0-6606d6-kserve-self-signed-certs\"" Nov 03 07:13:07.299315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.299294 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"openshift-service-ca.crt\"" Nov 03 07:13:07.299426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.299332 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"kube-root-ca.crt\"" Nov 03 07:13:07.326871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.326843 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:07.349952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.349923 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-tmp-dir\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.349952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.349965 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-dshm\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.350210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.349985 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2lk\" (UniqueName: \"kubernetes.io/projected/28861b2d-0779-4b84-bfff-76111f6068cd-kube-api-access-6v2lk\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.350210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.350116 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-home\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.350323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.350234 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28861b2d-0779-4b84-bfff-76111f6068cd-tls-certs\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.350323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.350278 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-model-cache\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.451329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.451221 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-model-cache\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.451517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.451348 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-tmp-dir\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.451517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.451398 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-dshm\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.451517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.451430 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2lk\" (UniqueName: \"kubernetes.io/projected/28861b2d-0779-4b84-bfff-76111f6068cd-kube-api-access-6v2lk\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.451517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.451463 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-home\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.451517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.451511 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28861b2d-0779-4b84-bfff-76111f6068cd-tls-certs\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.452637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.452603 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-home\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.452919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.452892 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-model-cache\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.453246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.453216 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-tmp-dir\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.456840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.455305 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-dshm\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.457511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.457485 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28861b2d-0779-4b84-bfff-76111f6068cd-tls-certs\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.526386 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.526351 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2lk\" (UniqueName: \"kubernetes.io/projected/28861b2d-0779-4b84-bfff-76111f6068cd-kube-api-access-6v2lk\") pod \"simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:07.822162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:07.822126 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:08.201944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:08.201922 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s"] Nov 03 07:13:08.204498 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:13:08.204455 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28861b2d_0779_4b84_bfff_76111f6068cd.slice/crio-47065a3ff4dc07b91967d84247803b8c6755fb3fc566a0b198e46ae7ce22fec9 WatchSource:0}: Error finding container 47065a3ff4dc07b91967d84247803b8c6755fb3fc566a0b198e46ae7ce22fec9: Status 404 returned error can't find the container with id 47065a3ff4dc07b91967d84247803b8c6755fb3fc566a0b198e46ae7ce22fec9 Nov 03 07:13:08.638597 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:08.638562 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" event={"ID":"28861b2d-0779-4b84-bfff-76111f6068cd","Type":"ContainerStarted","Data":"d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0"} Nov 03 07:13:08.638778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:08.638612 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" event={"ID":"28861b2d-0779-4b84-bfff-76111f6068cd","Type":"ContainerStarted","Data":"47065a3ff4dc07b91967d84247803b8c6755fb3fc566a0b198e46ae7ce22fec9"} Nov 03 07:13:08.800176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:08.800123 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podStartSLOduration=1.800110364 podStartE2EDuration="1.800110364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:13:08.799220554 +0000 UTC m=+848.732749211" watchObservedRunningTime="2026-11-03 07:13:08.800110364 +0000 UTC m=+848.733639020" Nov 03 07:13:08.822324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:08.822288 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:08.884456 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:08.884413 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:09.138114 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:13:09.138081 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:13:09.645290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:09.645255 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:13:09.657750 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:09.657712 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:13:10.657581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:10.657511 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:13:13.865128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:13.865084 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:16.090286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:16.090249 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:13:17.625967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:17.625936 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:17.823394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:17.823355 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:17.837144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:17.837114 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:18.709494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:18.709464 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:13:18.953055 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:18.952998 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:22.617163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:22.617117 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:23.946954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:23.946920 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:28.945965 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:28.945924 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:32.334272 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:32.334237 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:13:34.057814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.057779 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:34.157925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.157890 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm"] Nov 03 07:13:34.161781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.161764 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.229681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.229653 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm"] Nov 03 07:13:34.269169 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.269141 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"openshift-service-ca.crt\"" Nov 03 07:13:34.269169 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.269163 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"default-dockercfg-62d8x\"" Nov 03 07:13:34.269361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.269167 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"kube-root-ca.crt\"" Nov 03 07:13:34.269361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.269257 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"simulator-w6-3b41b9-kserve-self-signed-certs\"" Nov 03 07:13:34.307945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.307861 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-home\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.307945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.307901 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a870ab4-1945-477a-b018-1fe6fe7f1047-tls-certs\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.308189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.307981 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-dshm\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.308189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.308039 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-tmp-dir\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.308189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.308158 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zvh\" (UniqueName: \"kubernetes.io/projected/4a870ab4-1945-477a-b018-1fe6fe7f1047-kube-api-access-j4zvh\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.308189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.308188 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-model-cache\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.309957 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.309934 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:34.408935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.408900 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zvh\" (UniqueName: \"kubernetes.io/projected/4a870ab4-1945-477a-b018-1fe6fe7f1047-kube-api-access-j4zvh\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.408935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.408934 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-model-cache\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.409213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.409065 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-home\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.409213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.409119 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a870ab4-1945-477a-b018-1fe6fe7f1047-tls-certs\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.409213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.409167 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-dshm\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.409371 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.409216 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-tmp-dir\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.409421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.409386 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-model-cache\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.409421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.409404 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-home\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.409640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.409612 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-tmp-dir\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.411780 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.411746 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-dshm\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.411926 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.411908 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a870ab4-1945-477a-b018-1fe6fe7f1047-tls-certs\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.473860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.473827 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zvh\" (UniqueName: \"kubernetes.io/projected/4a870ab4-1945-477a-b018-1fe6fe7f1047-kube-api-access-j4zvh\") pod \"simulator-w6-3b41b9-kserve-8c97667bd-fh4wm\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:34.773109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:34.773073 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:35.152086 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:35.152063 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm"] Nov 03 07:13:35.153369 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:13:35.153342 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a870ab4_1945_477a_b018_1fe6fe7f1047.slice/crio-6ecfb970d9211049429e47d81513323cec257127423474bff1e5a4fc81ab19dd WatchSource:0}: Error finding container 6ecfb970d9211049429e47d81513323cec257127423474bff1e5a4fc81ab19dd: Status 404 returned error can't find the container with id 6ecfb970d9211049429e47d81513323cec257127423474bff1e5a4fc81ab19dd Nov 03 07:13:35.799504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:35.799468 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" event={"ID":"4a870ab4-1945-477a-b018-1fe6fe7f1047","Type":"ContainerStarted","Data":"383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984"} Nov 03 07:13:35.799708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:35.799516 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" event={"ID":"4a870ab4-1945-477a-b018-1fe6fe7f1047","Type":"ContainerStarted","Data":"6ecfb970d9211049429e47d81513323cec257127423474bff1e5a4fc81ab19dd"} Nov 03 07:13:35.930271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:35.930211 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podStartSLOduration=1.930196561 podStartE2EDuration="1.930196561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:13:35.928440223 +0000 UTC m=+875.861968892" watchObservedRunningTime="2026-11-03 07:13:35.930196561 +0000 UTC m=+875.863725216" Nov 03 07:13:36.773324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:36.773288 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:37.923022 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:37.922974 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:39.141070 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:13:39.141000 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:13:39.263834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:39.263802 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:44.391316 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.391280 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7"] Nov 03 07:13:44.394863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.394847 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.441074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.441039 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:44.505900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.505867 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"openshift-service-ca.crt\"" Nov 03 07:13:44.505900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.505878 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"kube-root-ca.crt\"" Nov 03 07:13:44.506175 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.506160 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"simulator-w1-9697f7-kserve-self-signed-certs\"" Nov 03 07:13:44.506231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.506190 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"default-dockercfg-5dnr4\"" Nov 03 07:13:44.508340 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.508321 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7"] Nov 03 07:13:44.513644 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.513622 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-tmp-dir\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.513749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.513733 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-dshm\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.513811 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.513769 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zjq\" (UniqueName: \"kubernetes.io/projected/dd5a47fd-67ac-479c-8064-b067ea059c6f-kube-api-access-p6zjq\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.513811 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.513793 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5a47fd-67ac-479c-8064-b067ea059c6f-tls-certs\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.513925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.513906 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-home\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.513987 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.513968 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-model-cache\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.614618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.614581 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-tmp-dir\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.614773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.614647 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-dshm\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.614773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.614669 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zjq\" (UniqueName: \"kubernetes.io/projected/dd5a47fd-67ac-479c-8064-b067ea059c6f-kube-api-access-p6zjq\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.614773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.614687 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5a47fd-67ac-479c-8064-b067ea059c6f-tls-certs\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.614773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.614711 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-home\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.614773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.614735 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-model-cache\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.615249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.615211 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-home\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.615369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.615244 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-model-cache\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.615369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.615278 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-tmp-dir\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.617612 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.617286 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-dshm\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.617612 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.617576 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5a47fd-67ac-479c-8064-b067ea059c6f-tls-certs\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.679242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.679169 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zjq\" (UniqueName: \"kubernetes.io/projected/dd5a47fd-67ac-479c-8064-b067ea059c6f-kube-api-access-p6zjq\") pod \"simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.690917 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.690880 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:44.707100 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.707074 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:44.773768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.773571 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:44.789320 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.789297 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:44.874834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:44.874803 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:13:45.109307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:45.109275 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7"] Nov 03 07:13:45.110144 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:13:45.110124 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd5a47fd_67ac_479c_8064_b067ea059c6f.slice/crio-d94bde3314ee37828b8a2ce726982af9fdec29e306b9144d56fa1e1d1ab32198 WatchSource:0}: Error finding container d94bde3314ee37828b8a2ce726982af9fdec29e306b9144d56fa1e1d1ab32198: Status 404 returned error can't find the container with id d94bde3314ee37828b8a2ce726982af9fdec29e306b9144d56fa1e1d1ab32198 Nov 03 07:13:45.866746 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:45.866698 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" event={"ID":"dd5a47fd-67ac-479c-8064-b067ea059c6f","Type":"ContainerStarted","Data":"b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387"} Nov 03 07:13:45.867276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:45.866758 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" event={"ID":"dd5a47fd-67ac-479c-8064-b067ea059c6f","Type":"ContainerStarted","Data":"d94bde3314ee37828b8a2ce726982af9fdec29e306b9144d56fa1e1d1ab32198"} Nov 03 07:13:45.991947 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:45.991891 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podStartSLOduration=1.991876273 podStartE2EDuration="1.991876273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:13:45.989891812 +0000 UTC m=+885.923420469" watchObservedRunningTime="2026-11-03 07:13:45.991876273 +0000 UTC m=+885.925404928" Nov 03 07:13:46.708483 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:46.708455 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:49.544540 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:49.544500 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:54.556155 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:54.556116 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:54.707805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:54.707769 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:54.720337 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:54.720313 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:54.932380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:54.932352 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:13:55.111497 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:55.111464 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:13:57.264866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:57.264782 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:13:59.565471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:13:59.565431 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:00.273816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:00.273789 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:14:00.276236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:00.276211 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:14:00.277631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:00.277609 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:14:00.280511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:00.280493 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:14:02.297782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:02.297745 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:04.542543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:04.542508 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:09.139065 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:14:09.139005 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:14:09.652782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:09.652739 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:14.740002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:14.739969 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:19.834812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:19.834775 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:24.839240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:24.839202 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:26.389820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:26.389786 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:14:28.214715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:28.214678 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:29.830720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:29.830677 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:34.894380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:34.894339 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:37.633994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.633959 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8"] Nov 03 07:14:37.637757 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.637740 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.731179 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.731148 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-home\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.731179 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.731180 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-dshm\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.731381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.731201 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-tmp-dir\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.731381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.731222 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tw2\" (UniqueName: \"kubernetes.io/projected/a28a39a6-785f-4b41-91ea-af2ac05b443f-kube-api-access-76tw2\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.731381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.731376 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28a39a6-785f-4b41-91ea-af2ac05b443f-tls-certs\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.731486 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.731401 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-model-cache\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.745496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.745468 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"premium-w0-6606d6-kserve-self-signed-certs\"" Nov 03 07:14:37.747944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.747926 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8"] Nov 03 07:14:37.774578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.774547 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:37.832307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832274 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-tmp-dir\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.832307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832314 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tw2\" (UniqueName: \"kubernetes.io/projected/a28a39a6-785f-4b41-91ea-af2ac05b443f-kube-api-access-76tw2\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.832555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832383 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28a39a6-785f-4b41-91ea-af2ac05b443f-tls-certs\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.832555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832418 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-model-cache\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.832555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832516 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-home\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.832555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832539 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-dshm\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.832790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832743 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-tmp-dir\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.832969 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.832946 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-model-cache\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.833085 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.833069 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-home\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.834951 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.834925 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-dshm\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.835345 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.835329 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28a39a6-785f-4b41-91ea-af2ac05b443f-tls-certs\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.898743 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.898673 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tw2\" (UniqueName: \"kubernetes.io/projected/a28a39a6-785f-4b41-91ea-af2ac05b443f-kube-api-access-76tw2\") pod \"premium-w0-6606d6-kserve-649d9c44f8-cpns8\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:37.949692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:37.949654 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:38.342523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:38.342500 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8"] Nov 03 07:14:38.344543 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:14:38.344515 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda28a39a6_785f_4b41_91ea_af2ac05b443f.slice/crio-3db549459368ded2873ddbd6060456c04b0d95c2c60ff573876e8a9677f37ac6 WatchSource:0}: Error finding container 3db549459368ded2873ddbd6060456c04b0d95c2c60ff573876e8a9677f37ac6: Status 404 returned error can't find the container with id 3db549459368ded2873ddbd6060456c04b0d95c2c60ff573876e8a9677f37ac6 Nov 03 07:14:39.137946 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:14:39.137914 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:14:39.192780 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:39.192738 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" event={"ID":"a28a39a6-785f-4b41-91ea-af2ac05b443f","Type":"ContainerStarted","Data":"0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda"} Nov 03 07:14:39.192982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:39.192795 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" event={"ID":"a28a39a6-785f-4b41-91ea-af2ac05b443f","Type":"ContainerStarted","Data":"3db549459368ded2873ddbd6060456c04b0d95c2c60ff573876e8a9677f37ac6"} Nov 03 07:14:39.313594 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:39.313542 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podStartSLOduration=2.313528437 podStartE2EDuration="2.313528437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:14:39.311856476 +0000 UTC m=+939.245385132" watchObservedRunningTime="2026-11-03 07:14:39.313528437 +0000 UTC m=+939.247057093" Nov 03 07:14:39.922065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:39.921974 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:39.950231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:39.950199 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:41.938582 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:41.938535 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:44.940109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:44.940057 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:47.950470 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:47.950438 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:47.963403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:47.963373 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:48.255679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:48.255604 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:14:48.455968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:48.455937 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:14:49.968911 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:49.968863 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:52.043046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:52.042986 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:54.959439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:54.959395 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:14:55.462180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:14:55.462134 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:00.035040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.034979 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:00.205938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.205904 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4"] Nov 03 07:15:00.209983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.209959 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:00.368288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.368251 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfck\" (UniqueName: \"kubernetes.io/projected/73283ae2-c034-4040-93c1-f91769b41072-kube-api-access-brfck\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:00.368499 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.368321 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73283ae2-c034-4040-93c1-f91769b41072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:00.378831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.378811 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:15:00.409331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.409299 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4"] Nov 03 07:15:00.469150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.469113 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73283ae2-c034-4040-93c1-f91769b41072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:00.469320 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.469272 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfck\" (UniqueName: \"kubernetes.io/projected/73283ae2-c034-4040-93c1-f91769b41072-kube-api-access-brfck\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:00.469765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.469744 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73283ae2-c034-4040-93c1-f91769b41072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:00.533385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.533353 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfck\" (UniqueName: \"kubernetes.io/projected/73283ae2-c034-4040-93c1-f91769b41072-kube-api-access-brfck\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:00.825467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:00.825436 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:01.205970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:01.205941 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4"] Nov 03 07:15:01.206504 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:15:01.206477 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73283ae2_c034_4040_93c1_f91769b41072.slice/crio-35c151151d0347f94e06c06d6300a090162a3e2867a1280bbc359c89ea295c21 WatchSource:0}: Error finding container 35c151151d0347f94e06c06d6300a090162a3e2867a1280bbc359c89ea295c21: Status 404 returned error can't find the container with id 35c151151d0347f94e06c06d6300a090162a3e2867a1280bbc359c89ea295c21 Nov 03 07:15:01.328871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:01.328832 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" event={"ID":"73283ae2-c034-4040-93c1-f91769b41072","Type":"ContainerStarted","Data":"35c151151d0347f94e06c06d6300a090162a3e2867a1280bbc359c89ea295c21"} Nov 03 07:15:05.064636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.064596 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:05.357787 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.357686 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" event={"ID":"73283ae2-c034-4040-93c1-f91769b41072","Type":"ContainerStarted","Data":"96b19502d82f5aa71dc38ec25b99700a3a1f257f57310eaf8023c1653104dc76"} Nov 03 07:15:05.480562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.480504 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" podStartSLOduration=1.883636555 podStartE2EDuration="5.48048976s" totalImagesPullingTime="3.596853205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:05.478364589 +0000 UTC m=+965.411893280" watchObservedRunningTime="2026-11-03 07:15:05.48048976 +0000 UTC m=+965.414018415" Nov 03 07:15:05.742880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.742795 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk"] Nov 03 07:15:05.747671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.747649 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:05.806770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.806737 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk"] Nov 03 07:15:05.854409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.854366 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"premium-w1-9697f7-kserve-self-signed-certs\"" Nov 03 07:15:05.906488 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.906435 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:05.926916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.926885 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-model-cache\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:05.927110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.926941 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkbh\" (UniqueName: \"kubernetes.io/projected/cef01dc3-a11f-461c-8d55-78763b38f002-kube-api-access-nqkbh\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:05.927110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.927051 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-tmp-dir\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:05.927110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.927082 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-dshm\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:05.927110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.927099 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef01dc3-a11f-461c-8d55-78763b38f002-tls-certs\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:05.927323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:05.927213 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-home\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.028613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.028516 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkbh\" (UniqueName: \"kubernetes.io/projected/cef01dc3-a11f-461c-8d55-78763b38f002-kube-api-access-nqkbh\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.028613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.028563 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-tmp-dir\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.028613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.028582 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-dshm\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.028613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.028607 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef01dc3-a11f-461c-8d55-78763b38f002-tls-certs\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.029137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.028651 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-home\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.029137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.028783 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-model-cache\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.029137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.029045 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-tmp-dir\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.029295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.029144 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-model-cache\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.029295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.029203 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-home\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.031460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.031437 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef01dc3-a11f-461c-8d55-78763b38f002-tls-certs\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.031576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.031468 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-dshm\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.094429 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.094386 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkbh\" (UniqueName: \"kubernetes.io/projected/cef01dc3-a11f-461c-8d55-78763b38f002-kube-api-access-nqkbh\") pod \"premium-w1-9697f7-kserve-bdd477bc7-kw6tk\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.122446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.122398 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:06.365078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.365041 4925 generic.go:356] "Generic (PLEG): container finished" podID="73283ae2-c034-4040-93c1-f91769b41072" containerID="96b19502d82f5aa71dc38ec25b99700a3a1f257f57310eaf8023c1653104dc76" exitCode=60 Nov 03 07:15:06.365281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.365087 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" event={"ID":"73283ae2-c034-4040-93c1-f91769b41072","Type":"ContainerDied","Data":"96b19502d82f5aa71dc38ec25b99700a3a1f257f57310eaf8023c1653104dc76"} Nov 03 07:15:06.365445 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.365422 4925 scope.go:122] "RemoveContainer" containerID="96b19502d82f5aa71dc38ec25b99700a3a1f257f57310eaf8023c1653104dc76" Nov 03 07:15:06.516643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:06.516615 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk"] Nov 03 07:15:06.518122 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:15:06.518094 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcef01dc3_a11f_461c_8d55_78763b38f002.slice/crio-24760eab9e1b276671dcb83204bebd2a19831b617d2c59f15ab8551842764fe8 WatchSource:0}: Error finding container 24760eab9e1b276671dcb83204bebd2a19831b617d2c59f15ab8551842764fe8: Status 404 returned error can't find the container with id 24760eab9e1b276671dcb83204bebd2a19831b617d2c59f15ab8551842764fe8 Nov 03 07:15:07.384119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:07.384065 4925 generic.go:356] "Generic (PLEG): container finished" podID="73283ae2-c034-4040-93c1-f91769b41072" containerID="0dfc780a77e4d48eed52819f1a21314208a4885504a803d552679007d497b650" exitCode=60 Nov 03 07:15:07.384695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:07.384101 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" event={"ID":"73283ae2-c034-4040-93c1-f91769b41072","Type":"ContainerDied","Data":"0dfc780a77e4d48eed52819f1a21314208a4885504a803d552679007d497b650"} Nov 03 07:15:07.384835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:07.384505 4925 scope.go:122] "RemoveContainer" containerID="0dfc780a77e4d48eed52819f1a21314208a4885504a803d552679007d497b650" Nov 03 07:15:07.384835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:07.384819 4925 scope.go:122] "RemoveContainer" containerID="96b19502d82f5aa71dc38ec25b99700a3a1f257f57310eaf8023c1653104dc76" Nov 03 07:15:07.385113 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:15:07.385072 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4_odh-ai-gateway-infra(73283ae2-c034-4040-93c1-f91769b41072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" podUID="73283ae2-c034-4040-93c1-f91769b41072" Nov 03 07:15:07.386988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:07.386965 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" event={"ID":"cef01dc3-a11f-461c-8d55-78763b38f002","Type":"ContainerStarted","Data":"922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf"} Nov 03 07:15:07.387133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:07.387001 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" event={"ID":"cef01dc3-a11f-461c-8d55-78763b38f002","Type":"ContainerStarted","Data":"24760eab9e1b276671dcb83204bebd2a19831b617d2c59f15ab8551842764fe8"} Nov 03 07:15:07.625861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:07.625811 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podStartSLOduration=2.625795467 podStartE2EDuration="2.625795467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:07.624616525 +0000 UTC m=+967.558145181" watchObservedRunningTime="2026-11-03 07:15:07.625795467 +0000 UTC m=+967.559324122" Nov 03 07:15:08.122883 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:08.122850 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:08.394694 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:08.394614 4925 scope.go:122] "RemoveContainer" containerID="0dfc780a77e4d48eed52819f1a21314208a4885504a803d552679007d497b650" Nov 03 07:15:08.395082 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:15:08.394856 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4_odh-ai-gateway-infra(73283ae2-c034-4040-93c1-f91769b41072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" podUID="73283ae2-c034-4040-93c1-f91769b41072" Nov 03 07:15:09.137459 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:15:09.137419 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:15:10.134347 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:10.134307 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:10.495601 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:10.495506 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:15.173142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:15.173100 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:16.122788 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:16.122754 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:16.135952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:16.135925 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:16.451655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:16.451567 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:15:20.295794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:20.295757 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:21.184505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:21.184471 4925 scope.go:122] "RemoveContainer" containerID="0dfc780a77e4d48eed52819f1a21314208a4885504a803d552679007d497b650" Nov 03 07:15:21.471330 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:21.471299 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" event={"ID":"73283ae2-c034-4040-93c1-f91769b41072","Type":"ContainerStarted","Data":"8ce172acfbc36c710d32ff4bce37fd889118d0203a60bac7e156a0fe31d70b89"} Nov 03 07:15:22.478691 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:22.478652 4925 generic.go:356] "Generic (PLEG): container finished" podID="73283ae2-c034-4040-93c1-f91769b41072" containerID="8ce172acfbc36c710d32ff4bce37fd889118d0203a60bac7e156a0fe31d70b89" exitCode=60 Nov 03 07:15:22.479113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:22.478721 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" event={"ID":"73283ae2-c034-4040-93c1-f91769b41072","Type":"ContainerDied","Data":"8ce172acfbc36c710d32ff4bce37fd889118d0203a60bac7e156a0fe31d70b89"} Nov 03 07:15:22.479113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:22.478780 4925 scope.go:122] "RemoveContainer" containerID="0dfc780a77e4d48eed52819f1a21314208a4885504a803d552679007d497b650" Nov 03 07:15:22.479113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:22.479107 4925 scope.go:122] "RemoveContainer" containerID="8ce172acfbc36c710d32ff4bce37fd889118d0203a60bac7e156a0fe31d70b89" Nov 03 07:15:22.479387 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:15:22.479363 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4_odh-ai-gateway-infra(73283ae2-c034-4040-93c1-f91769b41072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" podUID="73283ae2-c034-4040-93c1-f91769b41072" Nov 03 07:15:22.611549 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:22.611513 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4"] Nov 03 07:15:23.639532 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:23.639507 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:23.702649 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:23.702623 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73283ae2-c034-4040-93c1-f91769b41072-kube-api-access-brfck\" (UniqueName: \"kubernetes.io/projected/73283ae2-c034-4040-93c1-f91769b41072-kube-api-access-brfck\") pod \"73283ae2-c034-4040-93c1-f91769b41072\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " Nov 03 07:15:23.702800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:23.702680 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73283ae2-c034-4040-93c1-f91769b41072-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73283ae2-c034-4040-93c1-f91769b41072-maas-api-service-ca\") pod \"73283ae2-c034-4040-93c1-f91769b41072\" (UID: \"73283ae2-c034-4040-93c1-f91769b41072\") " Nov 03 07:15:23.703086 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:23.703056 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73283ae2-c034-4040-93c1-f91769b41072-maas-api-service-ca" pod "73283ae2-c034-4040-93c1-f91769b41072" (UID: "73283ae2-c034-4040-93c1-f91769b41072"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:15:23.704988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:23.704968 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73283ae2-c034-4040-93c1-f91769b41072-kube-api-access-brfck" pod "73283ae2-c034-4040-93c1-f91769b41072" (UID: "73283ae2-c034-4040-93c1-f91769b41072"). InnerVolumeSpecName "kube-api-access-brfck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:15:23.803400 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:23.803322 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfck\" (UniqueName: \"kubernetes.io/projected/73283ae2-c034-4040-93c1-f91769b41072-kube-api-access-brfck\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:15:23.803400 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:23.803347 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73283ae2-c034-4040-93c1-f91769b41072-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:15:24.497219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:24.497145 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" Nov 03 07:15:24.497219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:24.497183 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4" event={"ID":"73283ae2-c034-4040-93c1-f91769b41072","Type":"ContainerDied","Data":"35c151151d0347f94e06c06d6300a090162a3e2867a1280bbc359c89ea295c21"} Nov 03 07:15:24.497438 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:24.497249 4925 scope.go:122] "RemoveContainer" containerID="8ce172acfbc36c710d32ff4bce37fd889118d0203a60bac7e156a0fe31d70b89" Nov 03 07:15:24.672104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:24.672060 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4"] Nov 03 07:15:24.677878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:24.677843 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894835-wqfv4"] Nov 03 07:15:25.337564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:25.337523 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:26.191507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:26.191474 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73283ae2-c034-4040-93c1-f91769b41072" path="/var/lib/kubelet/pods/73283ae2-c034-4040-93c1-f91769b41072/volumes" Nov 03 07:15:30.358410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:30.358368 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:30.859665 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:30.859629 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:15:35.455042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:35.454976 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:39.138824 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:15:39.138781 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:15:40.532173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:40.532132 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:45.561054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:45.560986 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:50.637948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:50.637910 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:55.740906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:55.740858 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:15:55.980748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:15:55.980703 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:00.757405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:00.757312 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:05.775441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:05.775407 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:09.138458 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:16:09.138421 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:16:10.716896 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:10.716804 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:15.736979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:15.736929 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:20.372982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:20.372937 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:16:20.839307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:20.839257 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:25.957975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:25.957934 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:30.963348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:30.963310 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:36.047662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:36.047622 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:39.138140 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:16:39.138103 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:16:41.129892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:41.129852 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:45.270929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:45.270893 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:46.139118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:46.139065 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:47.487958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:47.487915 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:51.337724 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:51.337686 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:56.176515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:56.176476 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:56.412869 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:56.412830 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:16:57.862513 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:16:57.862477 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:17:01.652462 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:01.652423 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:06.610837 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:06.610792 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:09.140031 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:17:09.139966 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:17:11.642714 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:11.642680 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:16.842377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:16.842339 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:21.923544 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.923511 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7"] Nov 03 07:17:21.923932 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.923925 4925 cpu_manager.go:414] "containerMap: removing container" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.923973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.923936 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.923973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.923947 4925 cpu_manager.go:414] "containerMap: removing container" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.923973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.923952 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.924104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.924081 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.924104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.924088 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.927313 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.927296 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970056 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970106 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970123 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970241 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970273 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970291 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970406 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970448 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j566t\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-kube-api-access-j566t\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.970496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.970468 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:21.973803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.973776 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr"] Nov 03 07:17:21.974261 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.974245 4925 cpu_manager.go:414] "containerMap: removing container" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.974310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.974266 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.974369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.974359 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="73283ae2-c034-4040-93c1-f91769b41072" containerName="cleanup" Nov 03 07:17:21.977423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.977408 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:21.981484 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:21.981463 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6606d6-openshift-default-dockercfg-hxk2h\"" Nov 03 07:17:22.033632 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.033599 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:22.050437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.050412 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8"] Nov 03 07:17:22.054306 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.054283 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.072079 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072051 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.072203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072091 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.072203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072146 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.072289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072252 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.072347 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072331 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.072404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072389 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.072438 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072421 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.072485 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072456 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.072525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072498 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.072565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072537 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.072604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072573 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.072702 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072677 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004ae909-4d9d-4755-be60-cc0b651ebdcd-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.072793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072732 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.072793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072775 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.072902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072806 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.072902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072839 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.072902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072853 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.072902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072868 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.073143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.072923 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.073143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073130 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.073250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073167 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.073250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073173 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.073250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073194 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.073250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073237 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.073425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073266 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.073425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073344 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j566t\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-kube-api-access-j566t\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.073425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073382 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.073425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073409 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.073745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073435 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.073745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073541 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxg9\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-kube-api-access-smxg9\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.073745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073582 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.073745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.073626 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljr5\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-kube-api-access-7ljr5\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.075629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.075609 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.075629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.075619 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.081775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.081750 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr"] Nov 03 07:17:22.097417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.097396 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-669d6b-openshift-default-dockercfg-f5qrn\"" Nov 03 07:17:22.127360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.127334 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8"] Nov 03 07:17:22.136682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.136656 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.136813 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.136797 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j566t\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-kube-api-access-j566t\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.144716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.143340 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7"] Nov 03 07:17:22.174730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174653 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.174730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174715 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.174931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174746 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.174931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174779 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.174931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174876 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxg9\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-kube-api-access-smxg9\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.174931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174913 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174945 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljr5\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-kube-api-access-7ljr5\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.174984 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175005 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175045 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175091 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175148 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175174 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175205 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175210 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175293 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004ae909-4d9d-4755-be60-cc0b651ebdcd-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175341 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175388 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175416 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175463 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175209 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175536 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175794 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.175874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.175866 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.176357 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.176025 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.176436 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.176413 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004ae909-4d9d-4755-be60-cc0b651ebdcd-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.176436 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.176431 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.176711 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.176685 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.178122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.178097 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.178245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.178101 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.178740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.178718 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.178851 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.178833 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.180802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.180786 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1f2e7e-openshift-default-dockercfg-plh6k\"" Nov 03 07:17:22.239354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.239320 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.239496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.239454 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxg9\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-kube-api-access-smxg9\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.239568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.239545 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljr5\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-kube-api-access-7ljr5\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.239685 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.239669 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.244611 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.244596 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:22.291231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.291198 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:22.371680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.371647 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:22.680073 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:17:22.680005 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b404f1b_84ed_4ce1_99e8_9ae38a66e17f.slice/crio-a45e24b3cffdfb928da71cfd3c8735867c0873619f155db8fb3ae4b99da7ebfc WatchSource:0}: Error finding container a45e24b3cffdfb928da71cfd3c8735867c0873619f155db8fb3ae4b99da7ebfc: Status 404 returned error can't find the container with id a45e24b3cffdfb928da71cfd3c8735867c0873619f155db8fb3ae4b99da7ebfc Nov 03 07:17:22.680468 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.680444 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7"] Nov 03 07:17:22.682236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.682203 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.682345 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.682278 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.682345 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.682322 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.733978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.733954 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr"] Nov 03 07:17:22.735623 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:17:22.735600 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004ae909_4d9d_4755_be60_cc0b651ebdcd.slice/crio-323f258e124a412746c17ec9a194fa2ebb186d5f5a4641f2ccb2a994614a720d WatchSource:0}: Error finding container 323f258e124a412746c17ec9a194fa2ebb186d5f5a4641f2ccb2a994614a720d: Status 404 returned error can't find the container with id 323f258e124a412746c17ec9a194fa2ebb186d5f5a4641f2ccb2a994614a720d Nov 03 07:17:22.738800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.738768 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.738894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.738861 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.738963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.738948 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.785630 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.783879 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8"] Nov 03 07:17:22.785630 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:17:22.785080 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c152a1_8cbc_43c7_ba23_bb54f3ff2975.slice/crio-88e4449949cda8ae6a9b07be4c18bb860ef9149bb564ba3c7b482ccfdfebf194 WatchSource:0}: Error finding container 88e4449949cda8ae6a9b07be4c18bb860ef9149bb564ba3c7b482ccfdfebf194: Status 404 returned error can't find the container with id 88e4449949cda8ae6a9b07be4c18bb860ef9149bb564ba3c7b482ccfdfebf194 Nov 03 07:17:22.788832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.788798 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.788939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.788874 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:22.788939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:22.788917 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:17:23.210593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.209657 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" event={"ID":"15c152a1-8cbc-43c7-ba23-bb54f3ff2975","Type":"ContainerStarted","Data":"54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c"} Nov 03 07:17:23.210593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.209730 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" event={"ID":"15c152a1-8cbc-43c7-ba23-bb54f3ff2975","Type":"ContainerStarted","Data":"88e4449949cda8ae6a9b07be4c18bb860ef9149bb564ba3c7b482ccfdfebf194"} Nov 03 07:17:23.212566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.212065 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" event={"ID":"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f","Type":"ContainerStarted","Data":"ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874"} Nov 03 07:17:23.212566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.212139 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" event={"ID":"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f","Type":"ContainerStarted","Data":"a45e24b3cffdfb928da71cfd3c8735867c0873619f155db8fb3ae4b99da7ebfc"} Nov 03 07:17:23.213989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.213962 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" event={"ID":"004ae909-4d9d-4755-be60-cc0b651ebdcd","Type":"ContainerStarted","Data":"bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818"} Nov 03 07:17:23.214146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.214042 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" event={"ID":"004ae909-4d9d-4755-be60-cc0b651ebdcd","Type":"ContainerStarted","Data":"323f258e124a412746c17ec9a194fa2ebb186d5f5a4641f2ccb2a994614a720d"} Nov 03 07:17:23.245559 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.245524 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:23.247673 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.247634 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:23.247859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.247700 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 07:17:23.301035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.299710 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:23.301355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.300304 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:23.301497 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.301474 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 07:17:23.384071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.383390 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:23.407036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.385552 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 07:17:23.407036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.385619 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 07:17:23.477096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.475841 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podStartSLOduration=2.47582204 podStartE2EDuration="2.47582204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.348381401 +0000 UTC m=+1103.281910059" watchObservedRunningTime="2026-11-03 07:17:23.47582204 +0000 UTC m=+1103.409350698" Nov 03 07:17:23.645044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.642729 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podStartSLOduration=2.642709544 podStartE2EDuration="2.642709544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.497703078 +0000 UTC m=+1103.431231771" watchObservedRunningTime="2026-11-03 07:17:23.642709544 +0000 UTC m=+1103.576238196" Nov 03 07:17:23.645044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:23.644215 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podStartSLOduration=2.644205048 podStartE2EDuration="2.644205048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.641504777 +0000 UTC m=+1103.575033433" watchObservedRunningTime="2026-11-03 07:17:23.644205048 +0000 UTC m=+1103.577733707" Nov 03 07:17:24.245201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:24.245171 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:24.245773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:24.245746 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 07:17:24.296458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:24.296420 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:24.296731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:24.296703 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 07:17:24.375528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:24.375488 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 07:17:24.375815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:24.375772 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 07:17:25.245001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:25.244943 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:25.245001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:25.245004 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 07:17:25.313038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:25.297406 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:25.313038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:25.297468 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 07:17:25.374037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:25.373359 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 07:17:25.374037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:25.373417 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 07:17:26.244985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:26.244952 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:26.245240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:26.244999 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 07:17:26.295038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:26.294505 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:26.295038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:26.294583 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 07:17:26.376045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:26.375500 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 07:17:26.376045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:26.375582 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 07:17:27.249032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:27.248981 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:27.249663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:27.249628 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 07:17:27.319735 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:27.319694 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:27.321118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:27.321083 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 07:17:27.375550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:27.375498 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 07:17:27.375766 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:27.375584 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 07:17:27.724613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:27.724575 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:28.249154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:28.249113 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:28.249767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:28.249732 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 07:17:28.292707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:28.292674 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:28.293043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:28.292974 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 07:17:28.374557 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:28.374521 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 07:17:28.374892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:28.374838 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 07:17:29.245066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:29.245000 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:29.245276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:29.245103 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 07:17:29.292543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:29.292498 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:29.293082 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:29.292547 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 07:17:29.373196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:29.373158 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 07:17:29.373392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:29.373223 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 07:17:29.383091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:29.380427 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:17:30.299239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.299205 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:30.299644 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.299624 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:30.300430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.300409 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:17:30.377578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.377549 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:30.377861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.377840 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:30.378690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.378670 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:17:30.804376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.804347 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:30.804703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.804684 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:30.805651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:30.805634 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:17:32.930106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:32.930053 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:37.953545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:37.953471 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:39.138048 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:17:39.137983 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:17:40.365601 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:40.365330 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:42.677985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:42.677945 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:42.923113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:42.923080 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:43.128944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:43.128910 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:48.055474 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:48.055426 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:52.330769 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.330732 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn"] Nov 03 07:17:52.335172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.335151 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.439763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.439735 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn"] Nov 03 07:17:52.440101 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.440083 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ffee69\"/\"distinct-w4-ffee69-kserve-self-signed-certs\"" Nov 03 07:17:52.442847 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.442820 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9n4\" (UniqueName: \"kubernetes.io/projected/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-kube-api-access-zf9n4\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.442948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.442857 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-model-cache\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.442948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.442914 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-dshm\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.443047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.442992 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tmp-dir\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.443087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.443058 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tls-certs\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.443132 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.443108 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-home\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.472246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.472210 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:52.544483 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.544450 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-model-cache\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.544660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.544501 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-dshm\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.544660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.544609 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tmp-dir\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.544768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.544673 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tls-certs\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.544768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.544703 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-home\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.544978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.544953 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9n4\" (UniqueName: \"kubernetes.io/projected/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-kube-api-access-zf9n4\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.545111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.544988 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-model-cache\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.545111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.545064 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tmp-dir\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.545228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.545124 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-home\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.547967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.547941 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tls-certs\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.548190 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.548171 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-dshm\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.610458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.610372 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9n4\" (UniqueName: \"kubernetes.io/projected/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-kube-api-access-zf9n4\") pod \"distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:52.648200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:52.648167 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:53.054433 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:53.054333 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:53.064598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:53.064561 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn"] Nov 03 07:17:53.070130 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:17:53.070106 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b0ae12_0d99_4e6b_ab78_87ae0287b67e.slice/crio-c4bd447affaaa609965b420c2138758108da0e9cd126cb166d0725077e885993 WatchSource:0}: Error finding container c4bd447affaaa609965b420c2138758108da0e9cd126cb166d0725077e885993: Status 404 returned error can't find the container with id c4bd447affaaa609965b420c2138758108da0e9cd126cb166d0725077e885993 Nov 03 07:17:53.613600 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:53.613560 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" event={"ID":"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e","Type":"ContainerStarted","Data":"2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a"} Nov 03 07:17:53.614048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:53.613613 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" event={"ID":"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e","Type":"ContainerStarted","Data":"c4bd447affaaa609965b420c2138758108da0e9cd126cb166d0725077e885993"} Nov 03 07:17:53.649270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:53.649243 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:17:53.737224 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:53.737171 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podStartSLOduration=1.737158054 podStartE2EDuration="1.737158054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:53.736163353 +0000 UTC m=+1133.669692008" watchObservedRunningTime="2026-11-03 07:17:53.737158054 +0000 UTC m=+1133.670686710" Nov 03 07:17:54.108291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:54.108253 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:17:56.510163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:56.510121 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:58.170228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:58.170184 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:17:58.365146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:17:58.365109 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:02.649066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:02.649022 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:18:02.661759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:02.661732 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:18:02.680895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:02.680871 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:18:03.147315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:03.147275 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:06.801182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:06.801139 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:08.276368 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:08.276329 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:09.138741 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:18:09.138703 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:18:13.352044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:13.351976 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:13.959106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:13.958999 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:14.199170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:14.199131 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:15.760559 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.760520 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk"] Nov 03 07:18:15.764556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.764539 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.823358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.823328 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-dshm\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.823358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.823364 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-home\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.823609 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.823383 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-tmp-dir\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.823609 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.823485 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpz9\" (UniqueName: \"kubernetes.io/projected/a925815c-373f-4571-ba1a-0a0fbc93674a-kube-api-access-5qpz9\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.823736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.823667 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-model-cache\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.823736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.823709 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a925815c-373f-4571-ba1a-0a0fbc93674a-tls-certs\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.832089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.832062 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk"] Nov 03 07:18:15.871046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.871003 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6606d6\"/\"distinct-w0-6606d6-kserve-self-signed-certs\"" Nov 03 07:18:15.898024 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.897974 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:15.924523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.924492 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-dshm\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.924523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.924527 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-home\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.924767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.924546 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-tmp-dir\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.924767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.924674 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpz9\" (UniqueName: \"kubernetes.io/projected/a925815c-373f-4571-ba1a-0a0fbc93674a-kube-api-access-5qpz9\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.924899 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.924881 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-model-cache\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.924963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.924907 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-home\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.925097 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.925076 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-tmp-dir\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.925256 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.925226 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-model-cache\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.925333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.925307 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a925815c-373f-4571-ba1a-0a0fbc93674a-tls-certs\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.927131 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.927109 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-dshm\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.928180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.928162 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a925815c-373f-4571-ba1a-0a0fbc93674a-tls-certs\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:15.989543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:15.989510 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpz9\" (UniqueName: \"kubernetes.io/projected/a925815c-373f-4571-ba1a-0a0fbc93674a-kube-api-access-5qpz9\") pod \"distinct-w0-6606d6-kserve-58d55959c-t8kbk\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:16.076223 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:16.076182 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:16.483381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:16.483350 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk"] Nov 03 07:18:16.484284 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:18:16.484255 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda925815c_373f_4571_ba1a_0a0fbc93674a.slice/crio-0b0e7eb28656c83c61380fc2fd1e4c39ff0619f94eeb23849487397de1d9ff6e WatchSource:0}: Error finding container 0b0e7eb28656c83c61380fc2fd1e4c39ff0619f94eeb23849487397de1d9ff6e: Status 404 returned error can't find the container with id 0b0e7eb28656c83c61380fc2fd1e4c39ff0619f94eeb23849487397de1d9ff6e Nov 03 07:18:16.768727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:16.768633 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" event={"ID":"a925815c-373f-4571-ba1a-0a0fbc93674a","Type":"ContainerStarted","Data":"0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349"} Nov 03 07:18:16.768727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:16.768678 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" event={"ID":"a925815c-373f-4571-ba1a-0a0fbc93674a","Type":"ContainerStarted","Data":"0b0e7eb28656c83c61380fc2fd1e4c39ff0619f94eeb23849487397de1d9ff6e"} Nov 03 07:18:16.889935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:16.889883 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podStartSLOduration=1.889869562 podStartE2EDuration="1.889869562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:18:16.889292237 +0000 UTC m=+1156.822820903" watchObservedRunningTime="2026-11-03 07:18:16.889869562 +0000 UTC m=+1156.823398215" Nov 03 07:18:17.076587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:17.076561 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:18.343793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:18.343748 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:22.028338 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.028302 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg"] Nov 03 07:18:22.032247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.032223 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.087827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.087797 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3fea3a-openshift-default-dockercfg-9452b\"" Nov 03 07:18:22.095747 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.095722 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.095876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.095773 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.095925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.095883 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.095925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.095905 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d32cd9-4b79-46cc-b807-0e183522f514-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.096057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.095950 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2x9n\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-kube-api-access-m2x9n\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.096103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.096093 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.096138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.096122 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.096181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.096137 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.096181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.096162 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d32cd9-4b79-46cc-b807-0e183522f514-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.141004 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.140974 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg"] Nov 03 07:18:22.197674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.197642 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.197906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.197686 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.197906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.197729 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d32cd9-4b79-46cc-b807-0e183522f514-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.197906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.197781 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.197906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.197831 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.197951 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.197981 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d32cd9-4b79-46cc-b807-0e183522f514-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.198078 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2x9n\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-kube-api-access-m2x9n\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.198107 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.198142 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.198458 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198584 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.198559 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.198702 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.198635 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.199044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.199000 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d32cd9-4b79-46cc-b807-0e183522f514-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.200593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.200573 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.200674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.200632 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d32cd9-4b79-46cc-b807-0e183522f514-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.262363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.262324 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.262508 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.262338 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2x9n\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-kube-api-access-m2x9n\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.346136 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.346104 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:22.779934 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.779900 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg"] Nov 03 07:18:22.781098 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:18:22.781066 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d32cd9_4b79_46cc_b807_0e183522f514.slice/crio-68912ae549b1c59fca44533500467c56c2d91cae3a012fc7fb371ccdb5e75c08 WatchSource:0}: Error finding container 68912ae549b1c59fca44533500467c56c2d91cae3a012fc7fb371ccdb5e75c08: Status 404 returned error can't find the container with id 68912ae549b1c59fca44533500467c56c2d91cae3a012fc7fb371ccdb5e75c08 Nov 03 07:18:22.783391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.783357 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:18:22.783476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.783420 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:18:22.783476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.783446 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:18:22.807549 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:22.807525 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" event={"ID":"89d32cd9-4b79-46cc-b807-0e183522f514","Type":"ContainerStarted","Data":"68912ae549b1c59fca44533500467c56c2d91cae3a012fc7fb371ccdb5e75c08"} Nov 03 07:18:23.430101 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:23.430061 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:23.822072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:23.822027 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" event={"ID":"89d32cd9-4b79-46cc-b807-0e183522f514","Type":"ContainerStarted","Data":"dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c"} Nov 03 07:18:24.349160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:24.349127 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:24.349523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:24.349496 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 07:18:24.349686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:24.349661 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 07:18:25.364037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:25.363401 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 07:18:25.364037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:25.363460 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 07:18:26.077382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:26.077341 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:26.092737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:26.092694 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:26.218419 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:26.218368 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podStartSLOduration=5.218353706 podStartE2EDuration="5.218353706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:18:24.064303021 +0000 UTC m=+1163.997831693" watchObservedRunningTime="2026-11-03 07:18:26.218353706 +0000 UTC m=+1166.151882365" Nov 03 07:18:26.349510 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:26.349413 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 07:18:26.349762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:26.349726 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 07:18:26.882311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:26.882278 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:18:27.360416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:27.360377 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 07:18:27.360717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:27.360686 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 07:18:28.355039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:28.353396 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 07:18:28.355039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:28.353473 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 07:18:28.575736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:28.575699 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:29.347275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:29.347238 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 07:18:29.347471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:29.347312 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 07:18:30.352879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:30.352836 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:30.353289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:30.353238 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:30.353974 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:30.353953 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:18:30.506900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:30.506861 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:18:33.644266 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:33.644223 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:38.324986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:38.324942 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:38.658799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:38.658625 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:39.138267 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:18:39.138234 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:18:43.776109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:43.776064 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:48.730737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:48.730691 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:53.834038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:53.833976 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:54.104120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:54.104027 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:55.121904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:55.121855 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:56.857534 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:56.857499 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:18:58.862831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:18:58.862792 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:00.317436 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:00.317411 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:19:00.321044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:00.321004 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:19:00.322187 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:00.322163 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:19:00.326078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:00.326060 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:19:03.945913 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:03.945869 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:07.021751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.021714 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh"] Nov 03 07:19:07.025861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.025834 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.081874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.081836 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9697f7-openshift-default-dockercfg-pbdkl\"" Nov 03 07:19:07.132901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.132866 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.132922 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953d85fc-693d-4793-a125-83b802aef073-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.132963 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.132994 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953d85fc-693d-4793-a125-83b802aef073-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.133147 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.133170 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.133242 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.133269 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.133317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.133293 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7n5\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-kube-api-access-zj7n5\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.135712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.135682 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh"] Nov 03 07:19:07.234882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.234832 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.234900 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953d85fc-693d-4793-a125-83b802aef073-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.234934 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.234972 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953d85fc-693d-4793-a125-83b802aef073-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235350 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235165 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235350 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235192 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235361 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235382 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235400 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235471 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7n5\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-kube-api-access-zj7n5\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235610 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235658 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.235943 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.235919 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.236489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.236458 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953d85fc-693d-4793-a125-83b802aef073-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.237807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.237777 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953d85fc-693d-4793-a125-83b802aef073-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.237884 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.237844 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.300196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.300127 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.300377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.300352 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7n5\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-kube-api-access-zj7n5\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.339261 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.339232 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:07.350333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.350298 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:07.784790 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:19:07.784757 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953d85fc_693d_4793_a125_83b802aef073.slice/crio-e8c3c8f0e7364de687d7036e22b146419dcdc8c48166564c4e0e294f307b74b8 WatchSource:0}: Error finding container e8c3c8f0e7364de687d7036e22b146419dcdc8c48166564c4e0e294f307b74b8: Status 404 returned error can't find the container with id e8c3c8f0e7364de687d7036e22b146419dcdc8c48166564c4e0e294f307b74b8 Nov 03 07:19:07.784942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.784844 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh"] Nov 03 07:19:07.787544 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.787506 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:19:07.787633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.787603 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:19:07.787672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:07.787644 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:19:08.106830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:08.106795 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" event={"ID":"953d85fc-693d-4793-a125-83b802aef073","Type":"ContainerStarted","Data":"83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc"} Nov 03 07:19:08.107345 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:08.106862 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" event={"ID":"953d85fc-693d-4793-a125-83b802aef073","Type":"ContainerStarted","Data":"e8c3c8f0e7364de687d7036e22b146419dcdc8c48166564c4e0e294f307b74b8"} Nov 03 07:19:08.232200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:08.232137 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podStartSLOduration=2.232119696 podStartE2EDuration="2.232119696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:19:08.231410301 +0000 UTC m=+1208.164938958" watchObservedRunningTime="2026-11-03 07:19:08.232119696 +0000 UTC m=+1208.165648353" Nov 03 07:19:08.339358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:08.339325 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:08.340719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:08.340693 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 07:19:08.340848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:08.340740 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 07:19:09.063804 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:09.063756 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:09.139545 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:19:09.139508 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:19:09.340664 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:09.340571 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 07:19:09.340664 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:09.340629 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 07:19:10.339979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:10.339898 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 07:19:10.339979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:10.339953 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 07:19:11.340077 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:11.339998 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 07:19:11.340503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:11.340089 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 07:19:12.016068 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:12.016023 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:19:12.340206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:12.340167 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 07:19:12.340585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:12.340230 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 07:19:13.340246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:13.340204 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 07:19:13.340730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:13.340288 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 07:19:14.261323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:14.261279 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:14.340616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:14.340580 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 07:19:14.341132 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:14.340641 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 07:19:15.344807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:15.344767 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:15.345346 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:15.345100 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:15.345807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:15.345791 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:19:19.350335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:19.350298 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:19.769843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:19.769743 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:24.452369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:24.452333 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:29.571564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:29.571523 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:34.825709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:34.825672 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:39.142034 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:19:39.141972 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:19:39.980856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:39.980815 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:45.025886 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:45.025844 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:50.041111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:50.041067 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:55.138545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:55.138506 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:57.200823 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:57.200723 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:19:57.509273 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:19:57.509190 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:20:00.235243 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:00.235191 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:01.917594 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:01.917559 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:02.175128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:02.175040 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:05.349764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:05.349676 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:06.441031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:06.440955 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:09.139230 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:20:09.139188 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:20:10.425279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:10.425245 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:15.440655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:15.440615 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:20.455210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:20.455174 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:25.591348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:25.591301 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:30.668339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:30.668257 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:31.140620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:31.140586 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:35.082895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.082861 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t"] Nov 03 07:20:35.086872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.086850 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.144655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.144620 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-dshm\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.144837 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.144768 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-model-cache\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.144914 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.144837 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tls-certs\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.144982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.144955 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hgxb\" (UniqueName: \"kubernetes.io/projected/f5a95df8-8830-4852-af2b-ecf9b484bf3c-kube-api-access-7hgxb\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.145142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.145040 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-home\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.145203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.145167 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tmp-dir\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.190939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.190910 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t"] Nov 03 07:20:35.193820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.193793 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"distinct-2-w1-9697f7-kserve-self-signed-certs\"" Nov 03 07:20:35.245956 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.245916 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tls-certs\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.246161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.246036 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hgxb\" (UniqueName: \"kubernetes.io/projected/f5a95df8-8830-4852-af2b-ecf9b484bf3c-kube-api-access-7hgxb\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.246161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.246089 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-home\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.246161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.246151 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tmp-dir\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.246374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.246212 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-dshm\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.246374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.246275 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-model-cache\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.246809 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.246783 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-model-cache\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.247254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.247190 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tmp-dir\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.247523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.247503 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-home\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.250469 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.250446 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-dshm\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.250736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.250719 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tls-certs\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.253672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.253643 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:35.312425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.312388 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hgxb\" (UniqueName: \"kubernetes.io/projected/f5a95df8-8830-4852-af2b-ecf9b484bf3c-kube-api-access-7hgxb\") pod \"distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.399842 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.399758 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:35.759536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.759446 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:35.824139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:35.824113 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t"] Nov 03 07:20:35.824139 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:20:35.824129 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a95df8_8830_4852_af2b_ecf9b484bf3c.slice/crio-d0a9e35b4fb555201f91f6a7fb98a51c64cf27b50be811138f64bc0f32d63cee WatchSource:0}: Error finding container d0a9e35b4fb555201f91f6a7fb98a51c64cf27b50be811138f64bc0f32d63cee: Status 404 returned error can't find the container with id d0a9e35b4fb555201f91f6a7fb98a51c64cf27b50be811138f64bc0f32d63cee Nov 03 07:20:36.708341 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:36.708302 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" event={"ID":"f5a95df8-8830-4852-af2b-ecf9b484bf3c","Type":"ContainerStarted","Data":"6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5"} Nov 03 07:20:36.708748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:36.708354 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" event={"ID":"f5a95df8-8830-4852-af2b-ecf9b484bf3c","Type":"ContainerStarted","Data":"d0a9e35b4fb555201f91f6a7fb98a51c64cf27b50be811138f64bc0f32d63cee"} Nov 03 07:20:36.836863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:36.836814 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podStartSLOduration=1.836800206 podStartE2EDuration="1.836800206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:36.832883855 +0000 UTC m=+1296.766412511" watchObservedRunningTime="2026-11-03 07:20:36.836800206 +0000 UTC m=+1296.770328860" Nov 03 07:20:37.400120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:37.400088 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:39.138980 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:20:39.138948 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:20:40.849389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:40.849302 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:44.870437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:44.870403 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:45.399892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:45.399856 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:45.412662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:45.412635 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:45.779573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:45.779492 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:46.034981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:46.034892 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:51.247880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:51.247842 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:52.787823 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.787787 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k"] Nov 03 07:20:52.792147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.792112 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.904646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.904608 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k"] Nov 03 07:20:52.965501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965462 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.965642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965519 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.965707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965672 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.965747 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965733 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.965805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965791 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.965862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965842 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.965920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965908 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac500e48-730e-41f4-8521-0c7f53e7687f-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.965968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.965953 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:52.966067 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:52.966054 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdh5l\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-kube-api-access-fdh5l\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.067672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.067641 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.067838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.067700 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.067838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.067726 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.067838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.067755 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac500e48-730e-41f4-8521-0c7f53e7687f-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.067838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.067805 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068133 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068192 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdh5l\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-kube-api-access-fdh5l\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068302 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068337 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068478 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068414 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068531 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068789 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac500e48-730e-41f4-8521-0c7f53e7687f-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.068955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068934 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.069034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.068985 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.070504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.070475 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.071382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.071364 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.133715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.133680 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.137438 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.137412 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdh5l\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-kube-api-access-fdh5l\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.405434 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.405342 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:53.445091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.445059 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf"] Nov 03 07:20:53.450205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.450180 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.508998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.508954 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ffee69-openshift-default-dockercfg-mdk9q\"" Nov 03 07:20:53.512985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.512957 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf"] Nov 03 07:20:53.576198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576164 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576210 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbsr\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-kube-api-access-tjbsr\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576408 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576377 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576421 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576549 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576585 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86af9085-3448-4f1f-85fa-0d3585e307fe-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576660 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576688 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.576819 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.576800 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.677858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.677788 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.677858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.677825 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86af9085-3448-4f1f-85fa-0d3585e307fe-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.677858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.677849 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.677867 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.677925 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.677991 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678050 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbsr\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-kube-api-access-tjbsr\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678162 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678190 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678241 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678515 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678533 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678692 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86af9085-3448-4f1f-85fa-0d3585e307fe-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.678804 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.678781 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.680700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.680673 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.681151 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.681134 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.744163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.744134 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.744163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.744155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbsr\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-kube-api-access-tjbsr\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.782428 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.782401 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:53.867113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.862335 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:20:53.867113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.862421 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:20:53.867113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.862461 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:20:53.869507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:53.869347 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k"] Nov 03 07:20:54.111053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.111004 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:20:54.174643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.174608 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf"] Nov 03 07:20:54.187430 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:20:54.185559 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86af9085_3448_4f1f_85fa_0d3585e307fe.slice/crio-bd73d72c7c90f9814f0a00836bb6010037efb58539464a8f207d26e68eb2ec42 WatchSource:0}: Error finding container bd73d72c7c90f9814f0a00836bb6010037efb58539464a8f207d26e68eb2ec42: Status 404 returned error can't find the container with id bd73d72c7c90f9814f0a00836bb6010037efb58539464a8f207d26e68eb2ec42 Nov 03 07:20:54.190258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.190212 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:20:54.190360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.190306 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:20:54.190360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.190349 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:20:54.836034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.835595 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" event={"ID":"86af9085-3448-4f1f-85fa-0d3585e307fe","Type":"ContainerStarted","Data":"1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530"} Nov 03 07:20:54.836034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.835674 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" event={"ID":"86af9085-3448-4f1f-85fa-0d3585e307fe","Type":"ContainerStarted","Data":"bd73d72c7c90f9814f0a00836bb6010037efb58539464a8f207d26e68eb2ec42"} Nov 03 07:20:54.843035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.838099 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" event={"ID":"ac500e48-730e-41f4-8521-0c7f53e7687f","Type":"ContainerStarted","Data":"920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2"} Nov 03 07:20:54.843035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.838154 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" event={"ID":"ac500e48-730e-41f4-8521-0c7f53e7687f","Type":"ContainerStarted","Data":"f9e2d8eb442c0bd2c850376142093d68fd769a600cf1e1c254e404ef1d71b65f"} Nov 03 07:20:54.981033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:54.980433 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podStartSLOduration=1.98041077 podStartE2EDuration="1.98041077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:54.978361548 +0000 UTC m=+1314.911890204" watchObservedRunningTime="2026-11-03 07:20:54.98041077 +0000 UTC m=+1314.913939427" Nov 03 07:20:55.115094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:55.114139 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podStartSLOduration=3.114118248 podStartE2EDuration="3.114118248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:55.107681855 +0000 UTC m=+1315.041210514" watchObservedRunningTime="2026-11-03 07:20:55.114118248 +0000 UTC m=+1315.047646909" Nov 03 07:20:55.408092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:55.407558 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:20:55.408092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:55.407666 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:20:55.408092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:55.407707 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 07:20:55.797094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:55.787931 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:20:55.797094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:55.788067 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:20:55.797094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:55.788109 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 07:20:56.419040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.415166 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:56.419040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.416640 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:20:56.419040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.416685 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 07:20:56.465036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.464093 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:56.671189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.671086 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:56.711143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.711094 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:56.770146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.770100 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:56.785450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.785415 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:20:56.785703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.785677 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 07:20:56.842036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.841089 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:20:56.866037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:56.865619 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.081143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.081086 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.137658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.137610 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.314039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.313861 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.418223 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.418190 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:20:57.418491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.418465 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 07:20:57.455122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.455078 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.618261 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.618148 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.674367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.663717 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.675802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.675765 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:57.724385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.724327 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.725054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.725036 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:20:57.780032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.774170 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:57.780032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.774415 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:57.784385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.784359 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:20:57.784568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.784546 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 07:20:57.852137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.852094 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:20:57.852793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.852773 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:20:57.947136 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.931237 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:20:58.016635 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:57.993431 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:58.222208 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:58.222118 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:20:58.235347 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:58.235307 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:20:58.344736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:58.344704 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:20:58.416268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:58.416234 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:20:58.416505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:58.416477 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 07:20:58.793037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:58.790433 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:20:58.793037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:58.790495 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 07:20:59.038076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:59.038042 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:20:59.407038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:59.406439 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:20:59.407038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:59.406516 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 07:20:59.786508 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:59.786416 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:20:59.786778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:20:59.786748 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 07:21:00.409485 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:00.409445 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:00.410098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:00.410062 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 07:21:00.786529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:00.786436 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:00.786792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:00.786761 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 07:21:01.406291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:01.406253 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:01.406706 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:01.406673 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 07:21:01.783836 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:01.783744 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:01.784461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:01.784423 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 07:21:03.041076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:03.040992 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:21:03.041619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:03.041466 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:21:03.042623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:03.042596 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:21:03.250802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:03.250709 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:03.455256 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:03.455222 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:21:03.455519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:03.455495 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:04.074498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:04.074468 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:21:04.075749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:04.075726 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:21:04.076379 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:04.076361 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:21:08.355877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:08.355839 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:09.138489 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:21:09.138457 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:21:13.539598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:13.539512 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:18.542292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:18.542258 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:22.986219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:22.986175 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk"] Nov 03 07:21:22.991295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:22.991272 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.035154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.035123 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl"] Nov 03 07:21:23.038938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.038915 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl"] Nov 03 07:21:23.039107 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.039070 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.043204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.043185 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.125063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.125029 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl"] Nov 03 07:21:23.147290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.147256 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk"] Nov 03 07:21:23.151394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.151366 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr"] Nov 03 07:21:23.155952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.155925 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.159899 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.159871 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f264c5c8-6531-4981-bafe-61609a4a60fa-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.159911 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160111 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.160200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160164 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.160252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160201 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160301 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160269 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160297 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xcx\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-kube-api-access-b2xcx\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160324 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.160472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160351 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9l8v\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-kube-api-access-t9l8v\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.160472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160387 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160411 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24dm\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-kube-api-access-z24dm\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160446 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160483 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160535 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160560 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160616 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160640 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160680 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160703 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160730 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160779 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160811 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160836 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160875 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.160920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160904 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.161355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160950 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.161355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.160980 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f28e7d-5afd-4721-9700-d2e7163486cd-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.180589 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.180497 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl"] Nov 03 07:21:23.233946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.233911 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr"] Nov 03 07:21:23.262617 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262521 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.262617 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262576 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.262617 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262605 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262655 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262681 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262704 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262760 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262783 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262821 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262844 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.262894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262872 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262903 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262931 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262960 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262971 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.262985 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263028 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263079 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263108 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263153 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263180 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f28e7d-5afd-4721-9700-d2e7163486cd-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263223 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hssdl\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-kube-api-access-hssdl\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263270 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263295 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263340 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263355 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f264c5c8-6531-4981-bafe-61609a4a60fa-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263370 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263388 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.263409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263410 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263431 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263480 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263499 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xcx\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-kube-api-access-b2xcx\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263514 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263530 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9l8v\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-kube-api-access-t9l8v\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263555 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263573 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263588 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24dm\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-kube-api-access-z24dm\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263605 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263664 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.263893 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.264372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.264228 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.265036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.264775 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.265174 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.265155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.265416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.265398 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.265860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.265838 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.266056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.266033 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.266679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.266646 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f28e7d-5afd-4721-9700-d2e7163486cd-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.267023 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.266977 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.267264 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.267245 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.267541 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.267520 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f264c5c8-6531-4981-bafe-61609a4a60fa-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.268118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.268094 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.270866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.270847 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.271211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.271191 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.276512 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.276456 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.276692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.276667 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.277885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.277866 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.278354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.278175 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.333430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.333400 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.337257 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.337228 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.338265 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.338236 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.339101 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.339082 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xcx\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-kube-api-access-b2xcx\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.339740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.339715 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24dm\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-kube-api-access-z24dm\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.340003 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.339983 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9l8v\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-kube-api-access-t9l8v\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.356336 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.356313 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:23.364364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364338 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364389 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364417 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:23.364476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364459 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hssdl\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-kube-api-access-hssdl\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364500 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364525 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364729 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364781 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364828 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.364889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364878 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.365005 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.364963 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.365005 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.365028 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.365271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.365251 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.365342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.365325 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.365511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.365490 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.367423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.367397 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.367714 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.367692 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.430844 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.430691 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.432095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.431996 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hssdl\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-kube-api-access-hssdl\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.484455 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.484422 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:23.605092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.605034 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:23.639290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.639262 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:23.774946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.774915 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl"] Nov 03 07:21:23.777758 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.777722 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.777838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.777804 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.777878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.777844 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.829723 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:21:23.829697 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf603287_bda9_4061_bae6_a1dd53ec6dc4.slice/crio-8f8445c1ee61063ec3b1a0031b941ce9329230f66689e4dc08b80e4f14180220 WatchSource:0}: Error finding container 8f8445c1ee61063ec3b1a0031b941ce9329230f66689e4dc08b80e4f14180220: Status 404 returned error can't find the container with id 8f8445c1ee61063ec3b1a0031b941ce9329230f66689e4dc08b80e4f14180220 Nov 03 07:21:23.831598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.831575 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl"] Nov 03 07:21:23.832263 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.832232 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.832319 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.832296 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.832352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.832323 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.903898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.903871 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr"] Nov 03 07:21:23.911597 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:21:23.911564 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2166463c_54cc_4a4d_831d_2a4a72a3dacb.slice/crio-6c8c7487d4f8988c16bd3c8464f18de33973f00defece6adb115e95c040dc996 WatchSource:0}: Error finding container 6c8c7487d4f8988c16bd3c8464f18de33973f00defece6adb115e95c040dc996: Status 404 returned error can't find the container with id 6c8c7487d4f8988c16bd3c8464f18de33973f00defece6adb115e95c040dc996 Nov 03 07:21:23.915043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.914989 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.915121 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.915089 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:23.915165 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:23.915132 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:24.007678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.007648 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk"] Nov 03 07:21:24.008111 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:21:24.007916 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf264c5c8_6531_4981_bafe_61609a4a60fa.slice/crio-01bc86045cf83c92bc13f991a61f33d2701deac48d6a8697a80ce799eabe5d2b WatchSource:0}: Error finding container 01bc86045cf83c92bc13f991a61f33d2701deac48d6a8697a80ce799eabe5d2b: Status 404 returned error can't find the container with id 01bc86045cf83c92bc13f991a61f33d2701deac48d6a8697a80ce799eabe5d2b Nov 03 07:21:24.011001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.010962 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:24.011149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.011075 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:24.011149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.011119 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:24.214036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.212069 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" event={"ID":"bf603287-bda9-4061-bae6-a1dd53ec6dc4","Type":"ContainerStarted","Data":"83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5"} Nov 03 07:21:24.214036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.212127 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" event={"ID":"bf603287-bda9-4061-bae6-a1dd53ec6dc4","Type":"ContainerStarted","Data":"8f8445c1ee61063ec3b1a0031b941ce9329230f66689e4dc08b80e4f14180220"} Nov 03 07:21:24.220035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.215618 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" event={"ID":"44f28e7d-5afd-4721-9700-d2e7163486cd","Type":"ContainerStarted","Data":"cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce"} Nov 03 07:21:24.220035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.215667 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" event={"ID":"44f28e7d-5afd-4721-9700-d2e7163486cd","Type":"ContainerStarted","Data":"a934683e7922fd314184accafac5d1574b421cfa83ebdbf4c4a6777517529cf5"} Nov 03 07:21:24.220896 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.220247 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" event={"ID":"f264c5c8-6531-4981-bafe-61609a4a60fa","Type":"ContainerStarted","Data":"203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be"} Nov 03 07:21:24.220896 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.220300 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" event={"ID":"f264c5c8-6531-4981-bafe-61609a4a60fa","Type":"ContainerStarted","Data":"01bc86045cf83c92bc13f991a61f33d2701deac48d6a8697a80ce799eabe5d2b"} Nov 03 07:21:24.225160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.223920 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" event={"ID":"2166463c-54cc-4a4d-831d-2a4a72a3dacb","Type":"ContainerStarted","Data":"9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0"} Nov 03 07:21:24.225160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.223971 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" event={"ID":"2166463c-54cc-4a4d-831d-2a4a72a3dacb","Type":"ContainerStarted","Data":"6c8c7487d4f8988c16bd3c8464f18de33973f00defece6adb115e95c040dc996"} Nov 03 07:21:24.344420 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.344353 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podStartSLOduration=2.344339759 podStartE2EDuration="2.344339759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.340721355 +0000 UTC m=+1344.274250024" watchObservedRunningTime="2026-11-03 07:21:24.344339759 +0000 UTC m=+1344.277868414" Nov 03 07:21:24.358979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.358947 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:24.361127 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.361099 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:24.361336 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.361312 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:24.366185 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.366161 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:24.366435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.366417 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:24.366554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.366534 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:24.473579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.473527 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podStartSLOduration=2.473512556 podStartE2EDuration="2.473512556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.473027627 +0000 UTC m=+1344.406556282" watchObservedRunningTime="2026-11-03 07:21:24.473512556 +0000 UTC m=+1344.407041211" Nov 03 07:21:24.485908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.485877 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:24.487189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.487161 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:24.487337 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.487211 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:24.604425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.604323 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podStartSLOduration=2.604310414 podStartE2EDuration="2.604310414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.600961727 +0000 UTC m=+1344.534490384" watchObservedRunningTime="2026-11-03 07:21:24.604310414 +0000 UTC m=+1344.537839144" Nov 03 07:21:24.605267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.605238 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:24.730540 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:24.730486 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podStartSLOduration=1.7304726750000001 podStartE2EDuration="1.730472675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.728329588 +0000 UTC m=+1344.661858243" watchObservedRunningTime="2026-11-03 07:21:24.730472675 +0000 UTC m=+1344.664001404" Nov 03 07:21:25.375503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.375453 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.375988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.375519 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:25.376123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.376090 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.376307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.376275 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:25.492037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.489877 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.492037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.489953 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:25.618034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.614132 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.618034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:25.614192 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:26.249038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.248817 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.292198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.292150 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.358466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.358380 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:21:26.359080 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.359048 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.359226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.359100 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:26.377753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.377718 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.378600 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.378565 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:26.421165 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.421097 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.464182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.464120 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.465409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.464120 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.487332 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.487287 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.487522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.487390 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:26.605572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.605534 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.605885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.605841 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:26.617182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.617100 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.663173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.663127 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.698908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.698866 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.862095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.861248 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.930035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:26.928906 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:21:27.078154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.077230 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.210096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.205132 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.260131 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.260083 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.260783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.260757 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:21:27.314581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.314536 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.315228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.315204 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:21:27.371060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.359367 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:27.371060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.359427 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:27.371060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.359499 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.371060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.359812 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:21:27.474086 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.473268 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:21:27.474086 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.473731 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:21:27.549048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.512528 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:21:27.584168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.584129 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:21:27.878039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.865291 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:21:27.878039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.873212 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.962038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.936454 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.962845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.962814 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:21:27.963087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.963071 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:21:27.980297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:27.980255 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.122035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.090430 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:21:28.122035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.097899 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:21:28.122035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.097956 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:21:28.122035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.096371 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.122035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.098747 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:28.127941 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.126848 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.127941 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.126905 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:28.217095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.201626 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.217095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.201697 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:28.217095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.207448 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.320037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.271452 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.453052 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.362526 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:21:28.453052 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.363873 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.479391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.468128 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.479391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.468187 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:28.480057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.479995 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.495693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.495119 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.495693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.495176 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:28.496036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.495989 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.496194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.496170 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:28.563528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.563480 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.744096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.740239 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.750858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.748089 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:28.874134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.874088 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.910192 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.910150 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.005240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:28.997058 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.048204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.048148 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.106318 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.095511 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.127137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.127085 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.157198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.157146 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:21:29.157787 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.157752 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.280129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.280033 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:21:29.362033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.360990 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.362033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.361076 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:29.397651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.397609 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:21:29.399244 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.399213 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.423064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.420869 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.423064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.420928 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:29.505153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.505102 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.553622 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.553525 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:29.566381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.566340 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.579273 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.569876 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.579273 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.569930 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:29.610733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.610698 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.610984 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.610955 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:29.880774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.880113 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.948037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.944549 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:21:29.987536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:29.987505 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:21:30.093099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.057874 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.096410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.096370 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.096931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.096910 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:21:30.198240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.198135 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:21:30.228799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.198837 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:21:30.375039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.346289 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.440058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.437633 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.497103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.496536 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.497103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.496604 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:30.497103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.496689 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:21:30.656283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.656235 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.657797 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.657744 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.658041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.657999 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:30.683819 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.683774 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:21:30.717112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.689661 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.727150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.727104 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:30.727430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.712882 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.740040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.727562 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:30.748087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.745278 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.953078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.909723 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:21:30.956545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.956515 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:21:30.991135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:30.991089 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.184587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.184534 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 03 07:21:31.236060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.234087 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.269275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.269218 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.447631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.447513 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:21:31.448870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.448827 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.463046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.459421 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.463046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.459467 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:31.463046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.459530 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.463046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.459551 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:31.503064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.495137 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.503064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.495194 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:31.504071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.503444 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.504071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.503922 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:21:31.504071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.503965 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.621742 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.621689 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.621985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.621953 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:31.659781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.659707 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.886388 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.886343 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.911715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.910808 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.959200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.959114 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.993203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:31.993147 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.197528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.197423 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.238273 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.238229 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.270091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.270042 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:21:32.360432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.360396 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.361437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.361400 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:32.368834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.368801 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.369061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.369002 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:32.451262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.451149 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.452503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.452458 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.487465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.487432 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.487753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.487707 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:32.505143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.505089 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.623036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.619470 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.643294 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.643250 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:32.667543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.667496 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:21:32.744129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.723522 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:21:32.744440 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.744413 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:21:32.982472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.954211 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:21:32.982472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.973856 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.982472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.974261 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:21:32.995077 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:32.994193 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.007902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.007876 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:21:33.202145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.202101 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.240225 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.240106 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.270989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.270904 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:21:33.382040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.366052 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:21:33.382040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.368623 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.382040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.368670 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:33.382040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.368936 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.382040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.368968 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:33.395044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.394827 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:21:33.456459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.456410 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.486376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.486341 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.486666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.486638 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:33.618464 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.618354 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.618464 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.618407 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 07:21:33.686038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.685159 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:21:33.960222 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.960113 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:21:33.997529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:33.996573 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:21:34.043093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.035504 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.067328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.067299 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:21:34.109416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.109371 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:21:34.115990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.115964 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:21:34.135490 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.130564 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:21:34.398052 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.397998 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.401191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.401140 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 07:21:34.401409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.399428 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.401555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.401534 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 07:21:34.485451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.485414 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.485717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:34.485685 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:35.410815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.410788 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:35.438249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.423087 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:35.441961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.441890 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:21:35.487755 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.487719 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:21:35.487966 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.487776 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 07:21:35.528702 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.528670 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:35.530551 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.530526 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:35.536751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.536523 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:21:35.805093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.800614 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:35.988274 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.988220 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:35.989357 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.989333 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:35.991182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:35.991159 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:21:37.504473 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:37.504420 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:21:37.504970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:37.504482 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:37.573782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:37.573742 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:37.574338 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:37.574311 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:37.575546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:37.575527 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:21:38.040792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.040761 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb"] Nov 03 07:21:38.045751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.045734 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.079583 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079556 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.079713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079682 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.079827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079731 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.079874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079837 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.079874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079857 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.079956 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079873 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qch5\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-kube-api-access-5qch5\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.079956 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079894 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.080056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.079971 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.080099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.080061 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.107899 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.107875 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb"] Nov 03 07:21:38.181748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.181719 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.181748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.181750 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.181953 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.181773 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qch5\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-kube-api-access-5qch5\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.181998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.181980 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182033 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182110 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182181 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182217 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182258 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182326 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182377 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182539 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.182683 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182592 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.183004 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.182979 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.184823 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.184796 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.185437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.185412 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.247902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.247874 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.248182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.248156 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qch5\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-kube-api-access-5qch5\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.360363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.360317 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:38.803686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.803656 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb"] Nov 03 07:21:38.804608 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:21:38.804582 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5d0227_bbfb_4ff3_9b8f_2a57d4d838dd.slice/crio-3b22ba13105f074bb200bb9f86165af2cc4b6815e34ad228545fa01bca43af2c WatchSource:0}: Error finding container 3b22ba13105f074bb200bb9f86165af2cc4b6815e34ad228545fa01bca43af2c: Status 404 returned error can't find the container with id 3b22ba13105f074bb200bb9f86165af2cc4b6815e34ad228545fa01bca43af2c Nov 03 07:21:38.807063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.807030 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:38.807161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.807092 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:38.807161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:38.807121 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:21:39.148054 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:21:39.148000 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:21:39.301051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:39.299863 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" event={"ID":"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd","Type":"ContainerStarted","Data":"4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905"} Nov 03 07:21:39.301051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:39.299938 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" event={"ID":"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd","Type":"ContainerStarted","Data":"3b22ba13105f074bb200bb9f86165af2cc4b6815e34ad228545fa01bca43af2c"} Nov 03 07:21:39.361405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:39.361368 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:39.365149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:39.365119 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:21:39.365377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:39.365352 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 07:21:39.436564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:39.436452 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podStartSLOduration=2.436433191 podStartE2EDuration="2.436433191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:39.431204756 +0000 UTC m=+1359.364733416" watchObservedRunningTime="2026-11-03 07:21:39.436433191 +0000 UTC m=+1359.369961848" Nov 03 07:21:39.652989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:39.652945 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:40.382209 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:40.382165 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:21:40.382729 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:40.382231 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 07:21:41.171038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:41.165731 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:41.368034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:41.367583 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:21:41.368034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:41.367654 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 07:21:42.372037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:42.371416 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:21:42.372037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:42.371478 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 07:21:43.360767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:43.360718 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:21:43.361075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:43.361046 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 07:21:44.367688 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:44.367650 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:21:44.368307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:44.368275 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 07:21:45.361192 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:45.361153 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:21:45.361365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:45.361231 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 07:21:46.326902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:46.326847 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:46.367804 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:46.367773 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:46.369097 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:46.368727 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:46.370216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:46.370193 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:21:51.474916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:51.474872 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:53.150790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:53.150750 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:21:56.840030 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:56.839974 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:21:58.031488 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:21:58.031452 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:01.869749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:01.869697 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:06.971322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:06.971282 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:09.139465 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:22:09.139418 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:22:12.024328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:12.024106 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:14.470139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:14.470099 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:17.269303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:17.269269 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:22.433918 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:22.433880 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:24.244894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.230893 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w"] Nov 03 07:22:24.244894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.244189 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.289041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.288150 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.415172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.415126 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.447203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.447146 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:24.527107 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.522717 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w"] Nov 03 07:22:24.566205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.549195 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.567309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.567275 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.595799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.593257 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.596296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.596139 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.619674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.602050 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.619674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.602125 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.619674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.602197 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.619674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.602234 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.619674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.602260 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlvk\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-kube-api-access-pmlvk\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.619674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.602294 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.733731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.733699 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.748422 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.748393 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.749147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.749123 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.749658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.749638 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.776043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.770247 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.776043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.770339 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.776043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.770378 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.776043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.770405 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlvk\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-kube-api-access-pmlvk\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.776043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.770440 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.777441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.777372 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.777642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.748947 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.777642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.749564 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.778133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.748241 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.778280 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.750547 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.801572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.800589 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.842299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.842251 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.880234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.880171 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.894035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.883248 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlvk\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-kube-api-access-pmlvk\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:24.916608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:24.916573 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:25.021156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.021102 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.055121 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.022337 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.086035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.061442 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.129501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.129458 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.186038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.167587 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.188282 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.188247 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:25.292240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.291931 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:22:25.331180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.301139 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:22:25.452579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.438088 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:22:25.452579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.438488 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:22:25.483769 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.482624 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:25.486414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.484030 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.486414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.485471 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:22:25.486414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.486351 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Nov 03 07:22:25.631131 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.604765 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.656037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.631773 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:22:25.735133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.730130 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 03 07:22:25.803072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.802772 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.803072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.802867 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.894680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:25.894578 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:26.089663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.089607 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 07:22:26.089998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.089977 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:22:26.090609 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.090576 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:22:26.092502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.092464 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:26.170967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.103218 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:22:26.170967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.104350 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:22:26.170967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.104660 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:22:26.211064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.185496 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:22:26.211064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.185944 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:22:26.334915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.261515 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:22:26.334915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.298506 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:22:26.484120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.378712 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:22:26.484120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.440001 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:26.484120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.440397 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 03 07:22:26.526070 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.511659 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:26.592038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.567079 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:22:26.740866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:26.727635 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:22:27.133039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.129425 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.280110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.266221 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.280110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.267318 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.316455 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.297364 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.404027 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.403965 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:22:27.476155 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.404077 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.476155 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.463095 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.666977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.529375 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:27.666977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.590570 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.683448 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.683551 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.683599 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:22:27.684038 4925 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.323s" Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.684109 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.684163 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.684196 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.684232 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.684295 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:22:27.691196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.684313 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w"] Nov 03 07:22:27.697892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.697871 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:22:27.703226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.703206 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:22:27.713167 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.711935 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" event={"ID":"5524a86d-7cf2-4894-bdb7-6701d7f8a681","Type":"ContainerStarted","Data":"dd37f0a431eabd1cc2182c0aa52d2753ade8844fc839e6e24733e66570ef3f81"} Nov 03 07:22:27.732259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.732074 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:22:27.747955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.747885 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:22:27.753520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.753427 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:22:27.760839 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.758825 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:22:27.768110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:27.768091 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:22:28.730493 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:28.730464 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" event={"ID":"5524a86d-7cf2-4894-bdb7-6701d7f8a681","Type":"ContainerStarted","Data":"e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42"} Nov 03 07:22:29.190899 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:29.190863 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:29.191268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:29.191247 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 07:22:29.191418 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:29.191396 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 07:22:29.528317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:29.528236 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:29.710167 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:29.710111 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podStartSLOduration=5.710093048 podStartE2EDuration="5.710093048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:29.706884815 +0000 UTC m=+1409.640413470" watchObservedRunningTime="2026-11-03 07:22:29.710093048 +0000 UTC m=+1409.643621705" Nov 03 07:22:29.744207 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:29.744172 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:30.190257 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:30.190200 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 07:22:30.190574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:30.190535 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 07:22:30.379272 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:30.379235 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:31.189801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:31.189768 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 07:22:31.190250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:31.189826 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 07:22:32.189175 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:32.189138 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 07:22:32.189356 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:32.189218 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 07:22:33.189332 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:33.189296 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 07:22:33.189830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:33.189392 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 07:22:34.194450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:34.194416 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 07:22:34.194989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:34.194960 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 07:22:35.134659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:35.134623 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:35.499899 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:35.499795 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:35.500578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:35.500556 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:35.502182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:35.502163 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:22:38.085660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:38.085614 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:38.326452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:38.326408 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:38.533644 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:38.533555 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:39.140603 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:22:39.140568 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:22:39.599827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:39.599789 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:39.856491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:39.856415 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:40.054756 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:40.054722 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:40.334449 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:40.334398 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:40.651148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:40.635638 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:40.882930 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:40.882885 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:41.447688 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:41.447653 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:41.670754 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:41.669197 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:44.891602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:44.891559 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:45.137457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:45.137418 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:45.827216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:45.827168 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:46.016773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:46.016712 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:46.204089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:46.203971 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:46.602129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:46.602095 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:46.861066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:46.860958 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:48.611340 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:48.611295 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:48.846731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:48.846689 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:50.141276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:50.141228 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:50.339322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:50.339289 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:51.424546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:51.424510 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:53.675915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.675882 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg"] Nov 03 07:22:53.680361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.680333 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.708807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.708781 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:53.737256 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.737232 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz"] Nov 03 07:22:53.741732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.741711 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.788131 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.788100 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-3b41b9-openshift-default-dockercfg-zfqkd\"" Nov 03 07:22:53.800100 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.800061 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g"] Nov 03 07:22:53.805187 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.805093 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.805442 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.805419 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg"] Nov 03 07:22:53.825142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825096 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825159 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825207 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825305 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825449 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825486 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825562 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589ad64-54f2-4ae8-8844-351469ef11f2-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825628 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825612 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7c2\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-kube-api-access-bj7c2\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.825680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.825641 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.860753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.860700 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz"] Nov 03 07:22:53.901648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.901595 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g"] Nov 03 07:22:53.929956 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.928987 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589ad64-54f2-4ae8-8844-351469ef11f2-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.930269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.930246 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.930419 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.930398 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.930556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.930539 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7c2\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-kube-api-access-bj7c2\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.930655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.930641 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.930740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.930730 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.930816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.930806 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.930928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.929876 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589ad64-54f2-4ae8-8844-351469ef11f2-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.931288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.931151 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.931451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.931440 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.931599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.931584 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.931728 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.931718 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.932170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.932157 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.932344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.932325 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.932907 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.932886 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.933301 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933284 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.933405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.932826 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.933405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933229 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.933405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.932115 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.933525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933429 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.933525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933467 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.933649 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933631 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.933698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933668 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.933741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933715 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a8a718-f328-40ef-83e1-595a8cbba002-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.933807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933783 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.934296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933858 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.934296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.933991 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.934296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.934082 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.934296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.934138 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.934296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.934178 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6j2\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-kube-api-access-8m6j2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:53.934296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.934252 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcr2\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-kube-api-access-8rcr2\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:53.934296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.934258 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.937073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.937003 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.937398 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.937376 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:53.959270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:53.955298 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:54.013037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.000194 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:54.013037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.007364 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7c2\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-kube-api-access-bj7c2\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:54.035517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035475 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.035679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035524 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.035679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035551 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.035679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035590 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a8a718-f328-40ef-83e1-595a8cbba002-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.035679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035622 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.035679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035655 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.035916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035681 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.035916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035708 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6j2\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-kube-api-access-8m6j2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.035916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035758 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcr2\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-kube-api-access-8rcr2\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.035916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035820 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.035916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035870 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.035916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035901 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035931 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035966 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.035997 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036052 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036081 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036085 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036130 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.036281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036205 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.036873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036587 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.036873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036606 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.036873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036690 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a8a718-f328-40ef-83e1-595a8cbba002-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.036873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036758 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.036873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036848 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.037172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.036957 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.037506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.037481 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.037591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.037504 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.039138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.039115 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.040449 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.040428 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.041081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.041057 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.042083 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.042004 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.103430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.103390 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.104694 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.104652 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.104841 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.104711 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcr2\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-kube-api-access-8rcr2\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.104841 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.104811 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6j2\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-kube-api-access-8m6j2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.128707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.128676 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:54.332658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.297700 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:54.373423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.373389 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:54.595750 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.595718 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g"] Nov 03 07:22:54.599566 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:22:54.599541 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a8a718_f328_40ef_83e1_595a8cbba002.slice/crio-1d40a24e38038589aacc7e7da3cab774cb66ea8a235aa17da0968d0168a44626 WatchSource:0}: Error finding container 1d40a24e38038589aacc7e7da3cab774cb66ea8a235aa17da0968d0168a44626: Status 404 returned error can't find the container with id 1d40a24e38038589aacc7e7da3cab774cb66ea8a235aa17da0968d0168a44626 Nov 03 07:22:54.602289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.602256 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.602402 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.602322 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.602402 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.602349 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.799405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.799373 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg"] Nov 03 07:22:54.799771 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:22:54.799567 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda589ad64_54f2_4ae8_8844_351469ef11f2.slice/crio-ca39e4b78d91ca51ab356839a2a4f237e33669212e001ef75460f5b16e5a908b WatchSource:0}: Error finding container ca39e4b78d91ca51ab356839a2a4f237e33669212e001ef75460f5b16e5a908b: Status 404 returned error can't find the container with id ca39e4b78d91ca51ab356839a2a4f237e33669212e001ef75460f5b16e5a908b Nov 03 07:22:54.802102 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.802076 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.802160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.802136 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.802198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.802165 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.848468 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:22:54.848437 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f18c73_b9a7_4250_bb6e_2f88df5cb4bd.slice/crio-b67a566d7c84505427b8082c1d41f6863ecba99a8b5bbe386adb7da26005a228 WatchSource:0}: Error finding container b67a566d7c84505427b8082c1d41f6863ecba99a8b5bbe386adb7da26005a228: Status 404 returned error can't find the container with id b67a566d7c84505427b8082c1d41f6863ecba99a8b5bbe386adb7da26005a228 Nov 03 07:22:54.849180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.849158 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz"] Nov 03 07:22:54.851651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.851610 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.851765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.851693 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.851765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.851740 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:22:54.977260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.976268 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" event={"ID":"a589ad64-54f2-4ae8-8844-351469ef11f2","Type":"ContainerStarted","Data":"af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be"} Nov 03 07:22:54.977260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.976332 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" event={"ID":"a589ad64-54f2-4ae8-8844-351469ef11f2","Type":"ContainerStarted","Data":"ca39e4b78d91ca51ab356839a2a4f237e33669212e001ef75460f5b16e5a908b"} Nov 03 07:22:54.980036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.979447 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" event={"ID":"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd","Type":"ContainerStarted","Data":"2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65"} Nov 03 07:22:54.980036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.979516 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" event={"ID":"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd","Type":"ContainerStarted","Data":"b67a566d7c84505427b8082c1d41f6863ecba99a8b5bbe386adb7da26005a228"} Nov 03 07:22:54.982707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.982186 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" event={"ID":"a3a8a718-f328-40ef-83e1-595a8cbba002","Type":"ContainerStarted","Data":"122f67311791a48cc2df0cdad077977ef2bbd87a9bb8e0c7db64787c49008877"} Nov 03 07:22:54.982707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:54.982235 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" event={"ID":"a3a8a718-f328-40ef-83e1-595a8cbba002","Type":"ContainerStarted","Data":"1d40a24e38038589aacc7e7da3cab774cb66ea8a235aa17da0968d0168a44626"} Nov 03 07:22:55.113391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.113327 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podStartSLOduration=2.1133129840000002 podStartE2EDuration="2.113312984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.111547226 +0000 UTC m=+1435.045075882" watchObservedRunningTime="2026-11-03 07:22:55.113312984 +0000 UTC m=+1435.046841639" Nov 03 07:22:55.129721 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.129687 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:22:55.130984 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.130957 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:55.131156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.131032 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:22:55.258316 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.258241 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podStartSLOduration=2.258220512 podStartE2EDuration="2.258220512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.246413524 +0000 UTC m=+1435.179942181" watchObservedRunningTime="2026-11-03 07:22:55.258220512 +0000 UTC m=+1435.191749168" Nov 03 07:22:55.299063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.298952 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:22:55.300507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.300482 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:22:55.300634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.300532 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:22:55.375415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.375320 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:22:55.375757 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.375733 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:55.393832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.393765 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podStartSLOduration=2.393743726 podStartE2EDuration="2.393743726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.393427914 +0000 UTC m=+1435.326956573" watchObservedRunningTime="2026-11-03 07:22:55.393743726 +0000 UTC m=+1435.327272387" Nov 03 07:22:55.394155 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.375909 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:22:55.920062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:55.912841 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:56.135153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.135106 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:56.136144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.136111 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:22:56.165673 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.165638 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:56.300596 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.300488 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:22:56.300872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.300839 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:22:56.384554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.384518 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:56.384823 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.384795 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:22:56.835689 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:56.835649 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:57.133091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.132337 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:57.133091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.132411 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:22:57.299305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.298540 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:22:57.299305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.298592 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:22:57.300277 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.300232 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:57.384094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.380381 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:57.384094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.380447 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:22:57.463320 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.463278 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:57.535050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:57.517245 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:58.129072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.129037 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:58.129340 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.129313 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:22:58.305099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.305064 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:22:58.305679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.305644 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:22:58.388096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.377904 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:58.388096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.377975 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:22:58.497280 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.497228 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:58.603040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:58.602570 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:59.147054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.122547 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:22:59.235187 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.235117 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:59.235459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.235429 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:22:59.319233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.319199 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:22:59.319818 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.319785 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:22:59.420096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.394622 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:59.421791 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.421751 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:22:59.482145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.482100 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.512036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.508293 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.512036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.509040 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:22:59.534458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.534416 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.644781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:22:59.644737 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:00.215262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.215226 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:00.215539 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.215508 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:23:00.314351 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.314317 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:23:00.315605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.315569 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:23:00.598176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.501572 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.683038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.671088 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.782038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.698164 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.782459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.719256 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:00.782887 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.720318 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:00.783199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.783153 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:23:00.809304 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.737953 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:00.809559 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.737907 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.845089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.794905 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.855267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:00.794943 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:01.153717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.153630 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:01.260105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.260065 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:23:01.260352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.215738 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:01.406041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.405931 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:23:01.461231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.461190 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:23:01.593180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.593141 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:01.595135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.595099 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:23:01.658094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:01.654745 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:23:02.136128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:02.136074 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:02.136433 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:02.136398 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 07:23:02.298685 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:02.298648 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 07:23:02.298861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:02.298714 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 07:23:02.375503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:02.375467 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:02.375778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:02.375749 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 07:23:02.630037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:02.620812 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:03.492460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:03.492410 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:03.784094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:03.777992 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.384150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.384111 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:04.384450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.384415 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.450040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.448100 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:04.450040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.448161 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.463157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.463122 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:04.463430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.463398 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.513128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.473066 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:04.517325 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.517279 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.747537 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.747430 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:23:04.748079 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.748048 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:23:04.748521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.748496 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:04.748892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.748868 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 07:23:04.771107 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.751829 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:23:04.833234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:04.770156 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:05.132037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.130116 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:05.132037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.130183 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.261040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.240081 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.391203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.391105 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:05.391458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.391428 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.425042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.417340 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:23:05.426555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.426527 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:23:05.430414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.430393 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:23:05.518056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.509130 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:05.518056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.509205 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.518056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.517092 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:05.518808 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.518784 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:23:05.534067 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.520329 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.534775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.534748 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:23:05.701707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.701610 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:05.756980 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.756933 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:05.757667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.757642 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:23:05.759170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.759144 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.894796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.765179 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.895096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.765216 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:23:05.895580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.796657 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:23:05.895830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.895812 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:23:05.897082 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.797070 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.897277 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.805861 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.925059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.805919 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.925059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.805941 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Nov 03 07:23:05.925059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.912520 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:23:05.925059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.870323 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.925059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:05.913751 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:23:06.306297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.306249 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:23:06.319142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.319111 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:06.389034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.379524 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:06.403166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.403102 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.511063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.499128 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:06.511063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.499190 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.521461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.521428 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:06.522082 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.522048 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.567233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.567198 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:06.567502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.567472 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.599214 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.599170 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:06.702361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.676637 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.703548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.703513 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:06.729392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.729355 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:06.729652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.729622 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.785037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.759450 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.898745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.898640 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.899388 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.898970 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.918149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.918104 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.918759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.918730 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:06.927038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:06.919330 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:23:07.128036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.127591 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:07.128036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.127657 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.253524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.253420 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 03 07:23:07.286112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.282113 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:23:07.286394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.286379 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:23:07.536087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.535123 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:07.536087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.535191 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.625707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.624561 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:07.626700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.626672 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:07.626927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.626900 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.685111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.685068 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:23:07.698748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.698717 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:07.765202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.756778 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.765802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.765500 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.800954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.800850 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.917698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.917654 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.948038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:07.927134 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 07:23:08.112149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.105354 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:08.450099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.420271 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:23:08.608147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.608108 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:08.608768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.608726 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.626410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.626366 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.637940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.637902 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:08.661036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.638209 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.681852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.680147 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:08.681852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.680212 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.689130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.689085 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.704670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.702056 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.775076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.765673 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.775076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.766123 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:23:08.775076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.774689 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.814038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.812147 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.814038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.812573 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:23:08.843040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.839036 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.947039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.922430 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:23:08.947039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.929693 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.947039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.929629 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:23:09.015487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:08.952413 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.111871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.111820 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.113459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.112392 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:23:09.139932 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.139884 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:23:09.174516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.174478 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks"] Nov 03 07:23:09.212043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.200244 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.329870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.307362 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346298 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346374 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346420 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346535 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346565 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346665 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346768 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.369038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.346796 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.421042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.408175 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks"] Nov 03 07:23:09.489040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.469537 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.493285 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.493246 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.493499 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.493481 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.504094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.493800 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.504439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.504413 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.504626 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.504608 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.504884 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.504864 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.505033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.504995 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.505196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.505177 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnh9n\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-kube-api-access-bnh9n\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.509034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.504333 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.522041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.509894 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.522665 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.521973 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.522805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.522233 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.522903 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.522794 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.531248 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.527569 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:09.671880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.635999 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnh9n\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-kube-api-access-bnh9n\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.671880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.671084 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:09.671880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.671610 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.672506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.672456 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:09.672506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.672498 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.700737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.699137 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:09.700737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.699196 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.715066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.710146 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.729633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.719126 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.729633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.719415 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:23:09.750457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.745334 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:09.789200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.789156 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.802459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.789867 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:23:09.827026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.826967 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.936234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.936128 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 07:23:09.941670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.940098 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:23:09.941670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.941525 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:23:09.951048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:09.942590 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:23:10.015095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.004710 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:10.015095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.005306 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:10.130062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.116363 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:10.193115 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.162900 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnh9n\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-kube-api-access-bnh9n\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:10.412036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.401327 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:10.570089 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:23:10.482280 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:23:10.570089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.505179 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:10.603346 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.576455 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.752362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.717978 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:10.752362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.718079 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.752362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.718467 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Nov 03 07:23:10.752362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.718561 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.752362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.748509 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.839709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.830607 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.839709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.831441 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:10.839709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.831925 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:23:10.943708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.943677 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:23:10.943968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.943952 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:23:10.944172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.944156 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:23:10.944316 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.944304 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:23:10.944435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.944423 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:23:10.944555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.944544 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:23:10.957904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.957878 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:10.977166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:10.977139 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:23:11.024860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.024833 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:23:11.033322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.033301 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:23:11.034105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.034086 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:23:11.049205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.045640 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:23:11.049205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.047652 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:11.111096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.110466 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:23:11.120194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.120172 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:23:11.575765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.575735 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks"] Nov 03 07:23:11.576610 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:23:11.576582 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4098dcc2_2a51_4035_8c9c_1003fe447dd7.slice/crio-a8c5e10f9fe7bbcd628d2365361bf8e8093e110901457e111f53d9f43c56b6e4 WatchSource:0}: Error finding container a8c5e10f9fe7bbcd628d2365361bf8e8093e110901457e111f53d9f43c56b6e4: Status 404 returned error can't find the container with id a8c5e10f9fe7bbcd628d2365361bf8e8093e110901457e111f53d9f43c56b6e4 Nov 03 07:23:11.579418 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.579388 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:11.579536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.579509 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:11.579645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.579566 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:11.580121 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.580086 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:11.847069 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.846977 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:11.994348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.994306 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" event={"ID":"4098dcc2-2a51-4035-8c9c-1003fe447dd7","Type":"ContainerStarted","Data":"40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b"} Nov 03 07:23:11.994534 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:11.994373 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" event={"ID":"4098dcc2-2a51-4035-8c9c-1003fe447dd7","Type":"ContainerStarted","Data":"a8c5e10f9fe7bbcd628d2365361bf8e8093e110901457e111f53d9f43c56b6e4"} Nov 03 07:23:12.054408 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:12.054373 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:12.504942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:12.500797 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:12.505339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:12.505312 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:12.505518 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:12.505493 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 07:23:12.808562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:12.808448 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podStartSLOduration=4.80843097 podStartE2EDuration="4.80843097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:12.80316393 +0000 UTC m=+1452.736692585" watchObservedRunningTime="2026-11-03 07:23:12.80843097 +0000 UTC m=+1452.741959627" Nov 03 07:23:13.510040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:13.494686 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:13.510040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:13.494765 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 07:23:14.483800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:14.483764 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:14.484048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:14.483824 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 07:23:15.483552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:15.483514 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:15.484110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:15.483593 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 07:23:15.484942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:15.484736 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:15.700148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:15.700107 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:16.483520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:16.483474 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:16.483726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:16.483548 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 07:23:16.553034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:16.544944 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:16.957050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:16.956943 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:17.483565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:17.483521 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:17.483770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:17.483572 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 07:23:18.483824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:18.483785 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:18.484324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:18.483847 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 07:23:19.487597 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:19.487560 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:19.488262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:19.488231 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:19.488978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:19.488957 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:23:19.816908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:19.816867 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:20.074120 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:20.073979 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:20.789865 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:20.789826 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:21.131810 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:21.131781 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:21.327892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:21.327858 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:22.222385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:22.222350 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:27.639248 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:27.639211 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:28.234948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:28.234911 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:31.500459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:31.500418 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:32.687446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:32.687393 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:32.880502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:32.880466 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:33.078502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.078472 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:33.537782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.537701 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn"] Nov 03 07:23:33.543265 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.543245 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.637929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.637898 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn"] Nov 03 07:23:33.665665 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.665629 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e306e-987f-415d-994a-3d80a1673dff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.665858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.665675 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.665858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.665772 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85r7d\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-kube-api-access-85r7d\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.665858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.665840 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.666098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.665998 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.666098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.666088 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.666213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.666117 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.666213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.666138 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.666213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.666159 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e306e-987f-415d-994a-3d80a1673dff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.766930 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.766898 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e306e-987f-415d-994a-3d80a1673dff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767036 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e306e-987f-415d-994a-3d80a1673dff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767059 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767100 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85r7d\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-kube-api-access-85r7d\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767122 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767265 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767306 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767335 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767366 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767515 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767572 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767760 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767854 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e306e-987f-415d-994a-3d80a1673dff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.767910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.767854 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.769574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.769548 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e306e-987f-415d-994a-3d80a1673dff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.769753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.769737 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.832029 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.831985 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.832269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.832248 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85r7d\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-kube-api-access-85r7d\") pod \"maas-default-gateway-openshift-default-58d898bd55-zvjvn\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:33.858933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:33.858910 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:34.299366 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:23:34.299336 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e306e_987f_415d_994a_3d80a1673dff.slice/crio-b69609eca5875b3a2ff2ef8ec8630b04b7b928f68e5493b67831063a727a2dbe WatchSource:0}: Error finding container b69609eca5875b3a2ff2ef8ec8630b04b7b928f68e5493b67831063a727a2dbe: Status 404 returned error can't find the container with id b69609eca5875b3a2ff2ef8ec8630b04b7b928f68e5493b67831063a727a2dbe Nov 03 07:23:34.302524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:34.302480 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:34.302636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:34.302577 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:34.302636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:34.302622 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:34.303939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:34.303912 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn"] Nov 03 07:23:35.185153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:35.185118 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" event={"ID":"7e5e306e-987f-415d-994a-3d80a1673dff","Type":"ContainerStarted","Data":"bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c"} Nov 03 07:23:35.185610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:35.185171 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" event={"ID":"7e5e306e-987f-415d-994a-3d80a1673dff","Type":"ContainerStarted","Data":"b69609eca5875b3a2ff2ef8ec8630b04b7b928f68e5493b67831063a727a2dbe"} Nov 03 07:23:35.370035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:35.369400 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podStartSLOduration=2.369379011 podStartE2EDuration="2.369379011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:35.363244972 +0000 UTC m=+1475.296773631" watchObservedRunningTime="2026-11-03 07:23:35.369379011 +0000 UTC m=+1475.302907668" Nov 03 07:23:35.868035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:35.861422 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:35.868035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:35.863708 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:35.868035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:35.863765 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:36.864035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:36.863389 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:36.864035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:36.863453 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:37.869381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:37.869344 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:37.870031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:37.869976 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:38.005094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:38.005041 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:38.202845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:38.202768 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:38.399843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:38.399797 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:38.863376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:38.863338 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:38.863661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:38.863630 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:39.193741 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:23:39.193653 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:23:39.239489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.239454 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7"] Nov 03 07:23:39.252058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.249949 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.282958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.282530 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl"] Nov 03 07:23:39.288153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.287724 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.366407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.366363 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.366742 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.366719 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.367377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.367351 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.367546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.367528 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.367743 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.367724 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.367864 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.367849 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.367976 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.367962 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.368104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.368088 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.368217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.368203 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.368383 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.368365 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.368520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.368504 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.368640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.368622 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d907b61f-11a2-41fe-98b9-da8d93cc952d-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.368786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.368770 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.368990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.368972 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.373377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.373355 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrs5s\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-kube-api-access-wrs5s\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.373507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.373491 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.373651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.373636 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.373750 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.373736 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8rk\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-kube-api-access-hb8rk\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.385086 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.384952 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7"] Nov 03 07:23:39.414652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.414622 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl"] Nov 03 07:23:39.477515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.477416 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.478237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.478202 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8rk\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-kube-api-access-hb8rk\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.478605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.478582 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.478829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.478809 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.478927 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.478984 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479242 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479285 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479320 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479483 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479319 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479540 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479562 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479604 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.479786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479630 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.480431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.479967 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.480431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480145 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.480431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480233 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.480431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480277 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.480431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480314 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d907b61f-11a2-41fe-98b9-da8d93cc952d-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.480431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480366 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.480706 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480487 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.480706 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480542 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrs5s\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-kube-api-access-wrs5s\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.480706 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480587 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.481472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.480967 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.481472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.481247 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.481472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.481266 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.481816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.481795 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d907b61f-11a2-41fe-98b9-da8d93cc952d-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.482893 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.482874 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.483073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.483055 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.483344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.483325 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.483498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.483477 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.542213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.542183 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.543806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.543778 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8rk\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-kube-api-access-hb8rk\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.545139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.545097 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.554074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.553762 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrs5s\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-kube-api-access-wrs5s\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.586032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.585655 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:39.601890 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.601684 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:39.878969 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.878928 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:39.879186 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.878983 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:39.894368 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.894254 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb"] Nov 03 07:23:39.922150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.921208 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l"] Nov 03 07:23:39.922310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.922168 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.927383 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.927360 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.991130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.991090 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb"] Nov 03 07:23:39.991983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.991948 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992066 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992104 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992140 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992172 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992203 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992264 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkds\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-kube-api-access-slkds\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992307 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992366 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992401 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98ba875-0d62-41da-90d9-b8a8b8724e47-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992465 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992508 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c4453-3826-478d-987f-9f9d817dcd25-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.992681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992567 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992591 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992619 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992648 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992698 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktt6\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-kube-api-access-nktt6\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:39.992970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.992742 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:39.996332 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:39.996303 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l"] Nov 03 07:23:40.098495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098358 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.098495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098410 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.098681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098499 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.098681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098532 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.098681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098570 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.098681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098603 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.098681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098632 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.099037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098697 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkds\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-kube-api-access-slkds\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.099037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098745 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.099037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098811 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.099037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098846 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98ba875-0d62-41da-90d9-b8a8b8724e47-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.099037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098912 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.099037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.098954 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c4453-3826-478d-987f-9f9d817dcd25-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.105385 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.110356 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.114391 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.114952 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.115488 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.116361 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c4453-3826-478d-987f-9f9d817dcd25-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.119921 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.122764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.122660 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.129278 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:23:40.129195 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd907b61f_11a2_41fe_98b9_da8d93cc952d.slice/crio-1aea9391c2d9e5f2b14f89acad9cf1455296fb27b19ab0c59cd9000430b5641d WatchSource:0}: Error finding container 1aea9391c2d9e5f2b14f89acad9cf1455296fb27b19ab0c59cd9000430b5641d: Status 404 returned error can't find the container with id 1aea9391c2d9e5f2b14f89acad9cf1455296fb27b19ab0c59cd9000430b5641d Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134130 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134195 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134248 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134284 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134286 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134583 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134687 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.134837 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktt6\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-kube-api-access-nktt6\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.135096 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98ba875-0d62-41da-90d9-b8a8b8724e47-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.135423 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.136797 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.136877 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.139051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.136919 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.149052 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.145379 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.183434 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.173592 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl"] Nov 03 07:23:40.209685 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.209605 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktt6\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-kube-api-access-nktt6\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.212660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.211581 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.215564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.214486 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.215564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.214562 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.215564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.214604 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.215564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.215435 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.216688 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.216632 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkds\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-kube-api-access-slkds\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.222561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.221070 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7"] Nov 03 07:23:40.239189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.239159 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:40.245682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.245293 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:40.246166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.246142 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" event={"ID":"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1","Type":"ContainerStarted","Data":"d5ec6bd51048e531fb0499ea965f7e8183854fb9f4e5e3be2b6af668571e72d4"} Nov 03 07:23:40.248801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.248737 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" event={"ID":"d907b61f-11a2-41fe-98b9-da8d93cc952d","Type":"ContainerStarted","Data":"1aea9391c2d9e5f2b14f89acad9cf1455296fb27b19ab0c59cd9000430b5641d"} Nov 03 07:23:40.760357 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.760214 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb"] Nov 03 07:23:40.798945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.798601 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.798945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.798696 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.798945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.798746 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.860882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.860849 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:40.861075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.860926 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:40.931122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.931061 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l"] Nov 03 07:23:40.935281 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:23:40.935259 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98ba875_0d62_41da_90d9_b8a8b8724e47.slice/crio-da3114aa42a70aba497af3b220c2628b597e1872fa424c559d32e66d43244e37 WatchSource:0}: Error finding container da3114aa42a70aba497af3b220c2628b597e1872fa424c559d32e66d43244e37: Status 404 returned error can't find the container with id da3114aa42a70aba497af3b220c2628b597e1872fa424c559d32e66d43244e37 Nov 03 07:23:40.939661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.939634 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.939825 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.939810 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:40.939911 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:40.939900 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:23:41.432873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.432256 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" event={"ID":"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1","Type":"ContainerStarted","Data":"86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e"} Nov 03 07:23:41.436848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.436813 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" event={"ID":"d907b61f-11a2-41fe-98b9-da8d93cc952d","Type":"ContainerStarted","Data":"7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c"} Nov 03 07:23:41.442375 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.442345 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" event={"ID":"c98ba875-0d62-41da-90d9-b8a8b8724e47","Type":"ContainerStarted","Data":"27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568"} Nov 03 07:23:41.442527 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.442394 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" event={"ID":"c98ba875-0d62-41da-90d9-b8a8b8724e47","Type":"ContainerStarted","Data":"da3114aa42a70aba497af3b220c2628b597e1872fa424c559d32e66d43244e37"} Nov 03 07:23:41.665044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.649730 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:41.665044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.650155 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" event={"ID":"bb2c4453-3826-478d-987f-9f9d817dcd25","Type":"ContainerStarted","Data":"2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc"} Nov 03 07:23:41.665044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.650212 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" event={"ID":"bb2c4453-3826-478d-987f-9f9d817dcd25","Type":"ContainerStarted","Data":"6f7bc6c2a86eeba8ca925888e5871572c1866a0365e2f79d21a2eb099d118578"} Nov 03 07:23:41.701450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.690275 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:41.701450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.690425 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:41.701450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.690468 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:41.726036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.706463 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:41.726036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.706533 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:41.889034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.866323 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:41.889034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.866399 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:41.995156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:41.987239 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podStartSLOduration=2.987218591 podStartE2EDuration="2.987218591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:41.851615376 +0000 UTC m=+1481.785144037" watchObservedRunningTime="2026-11-03 07:23:41.987218591 +0000 UTC m=+1481.920747251" Nov 03 07:23:42.010059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.000255 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podStartSLOduration=3.000234544 podStartE2EDuration="3.000234544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:41.985048149 +0000 UTC m=+1481.918576806" watchObservedRunningTime="2026-11-03 07:23:42.000234544 +0000 UTC m=+1481.933763219" Nov 03 07:23:42.170041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.169810 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podStartSLOduration=3.169788166 podStartE2EDuration="3.169788166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:42.118523436 +0000 UTC m=+1482.052052084" watchObservedRunningTime="2026-11-03 07:23:42.169788166 +0000 UTC m=+1482.103316822" Nov 03 07:23:42.260096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.255350 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:42.260096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.255413 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:42.343817 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.343278 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:42.343817 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.343346 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:42.379037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.374187 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:42.379037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.374259 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:42.595037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.589891 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:42.595037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.589971 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:42.627036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.625376 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:42.627036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.625459 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:42.767559 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.767517 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:42.769160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.769103 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:42.832169 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.832106 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:42.871096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.862913 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:42.871607 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.871570 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 07:23:42.914176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.914131 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:42.914873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.914825 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:42.963663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:42.963613 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:43.040203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.040149 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:43.046145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.046099 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:43.052140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.052095 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:43.247663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.247557 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:43.247966 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.247909 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:43.252492 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.252464 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:43.252698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.252672 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:43.605171 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.605120 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:43.606163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.606025 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:43.606351 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.606326 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:43.606489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.606122 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:43.664037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.663137 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 03 07:23:43.778039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.768109 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:43.778039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.768586 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:23:43.806075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.799098 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:43.845814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.840114 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:43.845814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.840603 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:23:43.950093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.924103 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:43.950093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.924287 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:23:43.950503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.950461 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:43.951043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.951000 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:23:43.967148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.967097 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:23:43.967751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:43.967725 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:44.068785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.051369 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.068785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.051831 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:23:44.068785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.052916 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:23:44.068785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.053288 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:44.069254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.069214 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.069805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.069780 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:23:44.178448 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.168750 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podStartSLOduration=5.168728909 podStartE2EDuration="5.168728909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:42.288273913 +0000 UTC m=+1482.221802581" watchObservedRunningTime="2026-11-03 07:23:44.168728909 +0000 UTC m=+1484.102257567" Nov 03 07:23:44.268977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.254337 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:44.268977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.254398 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:44.268977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.254501 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:44.268977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.254529 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:44.600075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.591483 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:44.600075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.591541 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:44.633118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.606982 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:44.633118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.607060 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:44.652509 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.642419 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:44.816061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.798477 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.828670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.828634 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:44.855269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.828933 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.885739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.855532 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.885739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.861174 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:23:44.886603 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.886575 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:44.957123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.946170 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.983497 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.957529 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.983924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.975735 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:44.992330 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:44.991672 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:45.106162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.057192 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:45.106162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.075625 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:45.106162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.076524 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:45.106162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.076585 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:45.231037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.201719 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:45.282042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.253107 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:45.282042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.253168 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:45.282042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.274448 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:45.282042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.274505 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:45.651033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.649147 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:45.752074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.745174 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:23:45.783041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.768959 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:45.783041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.769066 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:45.783041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.769627 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:45.783041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.769664 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:45.822093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.812411 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:45.822448 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.822410 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:45.901088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.892440 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:23:45.901088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.894236 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:45.977036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.907611 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:45.977036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.907722 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:45.977036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:45.908103 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Nov 03 07:23:46.039355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.039295 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:46.081065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.064212 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 07:23:46.085036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.064270 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:23:46.197798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.112795 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:23:46.198618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.064325 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:23:46.198858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.124062 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:46.199123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.138132 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:46.199356 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.150971 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:46.199541 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.199514 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:46.305036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.304079 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:46.305036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.304141 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:46.305036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.304268 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:46.305036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.304316 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:46.508095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.493230 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 03 07:23:46.660047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.642806 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:46.660047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.642869 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:46.660047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.655095 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:46.703317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.703275 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:46.703962 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.703733 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:46.813133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.813094 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:46.894037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.890424 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:46.912038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.904666 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:46.912315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.906237 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:46.931925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.906286 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:46.931925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.920347 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:46.931925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.922447 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:46.995454 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:46.995423 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:23:47.083096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.079743 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:23:47.095075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.090651 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:47.115139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.115100 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:23:47.171414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.171371 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:47.212159 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.212092 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:47.212793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.212761 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:47.241236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.241197 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:47.241513 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.241482 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:47.258154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.258122 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:47.258826 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.258796 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:47.319038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.318482 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:47.440186 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.439841 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:23:47.650034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.588517 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:47.650034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.613073 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:47.650034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.627647 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:47.650034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.627703 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:47.675033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.656109 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:47.814097 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.813890 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:47.933038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.932174 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:23:47.935129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.935092 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:47.936631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:47.936288 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:48.083123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.081126 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:23:48.118058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.115390 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.118058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.116161 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.213536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.213488 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:48.215215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.215182 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:23:48.250434 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.250394 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:48.250705 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.250678 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:48.251097 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.251076 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:48.251278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.251257 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:48.467671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.467576 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.650566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.650526 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:48.650851 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.650818 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:48.727202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.727108 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.747373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.747340 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:48.747629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.747603 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:48.826398 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.825618 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.873851 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.873300 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.940586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.939841 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.940586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.940284 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:48.945195 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.945145 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:48.962310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:48.962227 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Nov 03 07:23:49.083124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.082350 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:49.116183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.116137 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:49.117269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.117232 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:49.216477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.216428 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:23:49.249041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.248372 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:49.249041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.248436 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:49.249599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.249576 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:49.249778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.249756 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:49.262545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.262503 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:49.480094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.477114 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:49.588575 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.588534 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 07:23:49.588892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.588844 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 07:23:49.609492 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.609459 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:49.609750 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.609724 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:49.630248 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.630221 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:49.737095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.724386 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:23:49.740829 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.738790 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:49.826216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.826159 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:49.946232 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.946169 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:49.950078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:49.946480 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:50.086197 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.086154 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:50.141036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.135687 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:50.166036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.157169 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:23:50.166036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.163164 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:50.270037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.245219 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:50.302173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.302129 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:50.379640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.379544 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:23:50.590038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.579369 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:50.590038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.579455 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:50.620036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.598559 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:23:50.652518 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.646480 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:50.656352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.653501 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:23:50.665384 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.665357 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:23:50.668031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.666459 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:50.668252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.666528 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:50.668447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.668423 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 07:23:50.731832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.731685 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:23:50.734518 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.734499 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:23:50.752676 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.752655 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:23:50.773026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.772990 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:23:50.958885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:50.958800 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:51.248252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.246439 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 07:23:51.248252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.246499 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 07:23:51.249312 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.249286 4925 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 07:23:51.281182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.281130 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 07:23:51.659043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.658987 4925 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:51.659372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.659339 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:51.809752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.809683 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:51.811343 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.811309 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:51.814746 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.814719 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:23:51.903095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.892879 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:51.903095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.894385 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:51.903095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:51.898563 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:23:52.314923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:52.308293 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:52.314923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:52.309647 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:52.314923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:52.310170 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:23:52.609538 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:52.608310 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:52.609538 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:52.609288 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:52.610213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:52.610192 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:23:55.999505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:55.999471 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:57.918038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:57.906125 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:57.918038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:57.911096 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:57.937136 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:57.937095 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.101865 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.101119 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 07:23:58.102695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.102126 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:58.102695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.102438 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 07:23:58.122030 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.121975 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:23:58.224212 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.224105 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.260038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.226456 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:23:58.777168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.777123 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.827381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.827127 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.913460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.911125 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.913460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.911702 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:23:58.913859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.913789 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.914577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.914558 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:23:58.945228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.945189 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:23:58.946895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.946863 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:23:58.955320 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:58.955233 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.108036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.107304 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.122037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.120269 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.122037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.120648 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:23:59.122037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.121716 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:23:59.122037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.121781 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:23:59.122037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.121817 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:23:59.305465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.251081 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:23:59.305465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.251498 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:23:59.342714 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.342670 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.352068 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.347109 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:23:59.526096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.520570 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:23:59.611954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.611911 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.915188 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:23:59.915137 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:24:00.035890 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.035826 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 07:24:00.037757 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.037725 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:24:00.112892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.112852 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:24:00.499943 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.499901 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 07:24:00.500879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.500849 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 07:24:00.555703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.543106 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:24:00.555703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.543500 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 07:24:00.780150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.779247 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Nov 03 07:24:00.821682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.821644 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 07:24:00.842524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.842437 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:24:00.855133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:00.829446 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Nov 03 07:24:01.044257 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.044162 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:24:01.117141 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.117081 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 07:24:01.254515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.254479 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:24:01.254712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.254546 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:24:01.254712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.254584 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:24:01.254712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.254624 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:24:01.292064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.272457 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:24:01.333070 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.331354 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:24:01.425572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.425455 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:24:01.431033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.427304 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:24:01.446366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.446346 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:24:01.451228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.451170 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:24:01.569556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.567839 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:24:01.573073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.572202 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:24:01.588104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.583407 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:24:01.588104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:01.585499 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:24:02.258039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:02.257974 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:07.236221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:07.236178 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:09.140321 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:24:09.140276 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:24:12.143938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:12.143888 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:12.387468 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:12.387433 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:12.598140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:12.598102 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:17.591722 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:17.591683 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:18.883501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.883463 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr"] Nov 03 07:24:18.887716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.887699 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942212 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942332 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942362 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942396 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pt68\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-kube-api-access-5pt68\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942441 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942499 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942550 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942586 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:18.942795 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:18.942753 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.043919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.043883 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.043935 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.043970 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044051 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044108 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044170 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044187 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044230 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pt68\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-kube-api-access-5pt68\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044256 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044567 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044352 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044567 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044410 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.044915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044853 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.045098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.044971 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.045355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.045327 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.046796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.046774 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.047002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.046981 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.105923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.105884 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr"] Nov 03 07:24:19.113956 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.113890 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pt68\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-kube-api-access-5pt68\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.114598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.114550 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmkwr\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.204496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.204412 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:19.645281 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:24:19.645245 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43682d6b_e6ca_4ea0_be2f_91dc22fe735c.slice/crio-c98b2a90389d2606be894619e8052a2dbf9e228ed51c461fc1d3f3c28fb02e3c WatchSource:0}: Error finding container c98b2a90389d2606be894619e8052a2dbf9e228ed51c461fc1d3f3c28fb02e3c: Status 404 returned error can't find the container with id c98b2a90389d2606be894619e8052a2dbf9e228ed51c461fc1d3f3c28fb02e3c Nov 03 07:24:19.647701 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.647672 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr"] Nov 03 07:24:19.648175 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.648147 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:24:19.648230 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.648214 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:24:19.648271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:19.648249 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:24:20.307161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:20.307126 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" event={"ID":"43682d6b-e6ca-4ea0-be2f-91dc22fe735c","Type":"ContainerStarted","Data":"e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa"} Nov 03 07:24:20.307553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:20.307206 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" event={"ID":"43682d6b-e6ca-4ea0-be2f-91dc22fe735c","Type":"ContainerStarted","Data":"c98b2a90389d2606be894619e8052a2dbf9e228ed51c461fc1d3f3c28fb02e3c"} Nov 03 07:24:20.489840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:20.489783 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podStartSLOduration=2.4897689659999998 podStartE2EDuration="2.489768966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:20.485963723 +0000 UTC m=+1520.419492378" watchObservedRunningTime="2026-11-03 07:24:20.489768966 +0000 UTC m=+1520.423297692" Nov 03 07:24:21.205137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:21.205103 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:21.206412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:21.206390 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 07:24:21.206508 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:21.206436 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 07:24:22.205163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:22.205128 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 07:24:22.205568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:22.205190 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 07:24:22.884059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:22.881090 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:23.205718 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:23.205637 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 07:24:23.205718 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:23.205703 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 07:24:24.205425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:24.205385 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 07:24:24.205595 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:24.205445 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 07:24:25.205324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:25.205291 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 07:24:25.205708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:25.205358 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 07:24:26.205513 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:26.205478 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 07:24:26.205905 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:26.205534 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 07:24:27.204826 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:27.204791 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 07:24:27.204996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:27.204852 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 07:24:28.145997 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:28.145958 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:28.212814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:28.212775 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:28.213255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:28.213234 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:28.214083 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:28.214060 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:24:33.334453 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:33.334416 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:33.968484 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:33.968449 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:24:34.016921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.016889 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8"] Nov 03 07:24:34.021514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.021497 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.101701 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.101749 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.101801 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.101853 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh7vn\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-kube-api-access-kh7vn\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.101918 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.101944 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.101969 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.102090 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.102503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.102250 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.144812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.144771 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8"] Nov 03 07:24:34.203335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203301 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203333 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203350 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203407 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203506 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203761 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203648 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203761 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203678 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203761 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203741 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203773 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203791 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.203929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.203804 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh7vn\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-kube-api-access-kh7vn\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.204109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.204086 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.204565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.204298 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.204565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.204511 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.206173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.206144 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.206279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.206252 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.270115 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.269986 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.272916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.272891 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh7vn\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-kube-api-access-kh7vn\") pod \"maas-default-gateway-openshift-default-58d898bd55-fvch8\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:34.338418 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:34.338393 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:35.011756 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:35.011722 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8"] Nov 03 07:24:35.032774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:35.032734 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:24:35.033033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:35.032996 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:24:35.033145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:35.033108 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:24:35.436529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:35.436491 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" event={"ID":"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0","Type":"ContainerStarted","Data":"abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27"} Nov 03 07:24:35.437035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:35.436551 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" event={"ID":"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0","Type":"ContainerStarted","Data":"97a01ecfc06c2c7ab4cae800d80eb1e8624e0216174964cd663f8c7db37aae1c"} Nov 03 07:24:35.565034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:35.564963 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podStartSLOduration=2.564949358 podStartE2EDuration="2.564949358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:35.561887457 +0000 UTC m=+1535.495416114" watchObservedRunningTime="2026-11-03 07:24:35.564949358 +0000 UTC m=+1535.498478091" Nov 03 07:24:36.339157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:36.339124 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:36.340485 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:36.340453 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:24:36.340609 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:36.340519 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 07:24:36.799048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:36.798907 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:37.008423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:37.008381 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:37.339645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:37.339604 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:24:37.339851 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:37.339686 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 07:24:38.339129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:38.339088 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:24:38.339530 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:38.339156 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 07:24:38.894039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:38.893464 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:39.139746 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:24:39.139696 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:24:39.339579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:39.339548 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:24:39.339950 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:39.339610 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 07:24:40.339365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:40.339328 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:24:40.339517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:40.339386 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 07:24:41.339511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:41.339479 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:24:41.339904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:41.339531 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 07:24:42.339163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:42.339126 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:24:42.339396 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:42.339194 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 07:24:43.702893 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:43.702861 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:43.703475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:43.703438 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:43.704298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:43.704271 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:24:44.275160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:44.275115 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:47.706469 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:47.706420 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:49.890917 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:49.890878 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:55.200040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:55.199984 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:55.631211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:55.631175 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:55.822580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:55.822530 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:24:59.388937 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:24:59.388904 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:00.295260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:00.295221 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:00.590671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:00.590631 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:00.779847 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:00.779793 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:02.518362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:02.518327 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:05.436576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:05.436538 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:09.140069 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:25:09.140032 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:25:09.459777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:09.459690 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:10.551265 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:10.551161 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:15.655141 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:15.655097 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:21.219461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:21.219418 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:26.438288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:26.438192 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:30.850486 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:30.850406 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:31.687612 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:31.687576 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:31.952648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:31.952560 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:32.159279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:32.159243 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:32.459664 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:32.459631 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:32.750488 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:32.750401 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:33.288036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:33.287982 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:33.795265 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:33.795229 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:33.992414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:33.992383 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:34.187690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:34.187643 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:34.419962 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:34.419918 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:34.621991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:34.620485 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:35.061142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:35.056554 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:35.256901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:35.256861 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:36.469406 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:36.469377 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:36.759415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:36.759322 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:38.556380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:38.556277 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:38.762124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:38.762081 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:38.963923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:38.963877 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:39.142520 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:25:39.142460 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:25:39.862427 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:39.862380 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:41.391116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:41.391081 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:41.589658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:41.589619 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:41.795561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:41.795431 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:41.998738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:41.998704 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:42.254065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:42.252137 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:45.430536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:45.430497 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:45.494283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:45.494245 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:25:47.063048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:47.061989 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:47.279084 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:47.278943 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:47.486088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:47.482536 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:49.462901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:49.462857 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:25:49.464512 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:49.464481 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 07:25:49.553129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:49.553087 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:25:49.778203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:49.776031 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:25:51.055078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:51.055042 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:56.096487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:56.096449 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:59.639803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:59.639761 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:25:59.875895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:25:59.875861 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:00.459405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:00.459370 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:02.106605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:02.106563 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:02.337127 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:02.337093 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:02.528812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:02.528722 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:04.155002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:04.154969 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:05.109833 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:05.109797 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:05.368822 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:05.368735 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:05.565417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:05.565382 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:07.046860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:07.046819 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:07.736530 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:07.736455 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:09.142141 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:26:09.142107 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:26:09.376565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.376535 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs"] Nov 03 07:26:09.380321 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.380300 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.436043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.435953 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 07:26:09.436200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.436091 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 07:26:09.436331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.436306 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 07:26:09.483156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.483124 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.483333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.483209 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.483333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.483299 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.483452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.483371 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvn5w\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-kube-api-access-rvn5w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.483452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.483401 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.483637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.483606 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.483773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.483737 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9595de-7246-4ec1-8b88-b2286d081ebb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.485892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.485861 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zrhnm\"" Nov 03 07:26:09.492749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.492713 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs"] Nov 03 07:26:09.585629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.585590 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.585814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.585696 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvn5w\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-kube-api-access-rvn5w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.585814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.585728 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.586053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.585884 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.586053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.585988 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9595de-7246-4ec1-8b88-b2286d081ebb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.586191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.586087 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.586250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.586189 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.587209 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.587184 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.588564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.588537 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.589484 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.589453 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9595de-7246-4ec1-8b88-b2286d081ebb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.589581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.589492 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.589581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.589552 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.650135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.650105 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvn5w\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-kube-api-access-rvn5w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.650264 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.650153 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrngs\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:09.693988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:09.693914 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:10.136910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:10.136877 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs"] Nov 03 07:26:10.137060 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:26:10.136931 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9595de_7246_4ec1_8b88_b2286d081ebb.slice/crio-9532c2a7e9c4ca27deb0c80695f7b401ae167d52dd977969baf135043bed2a28 WatchSource:0}: Error finding container 9532c2a7e9c4ca27deb0c80695f7b401ae167d52dd977969baf135043bed2a28: Status 404 returned error can't find the container with id 9532c2a7e9c4ca27deb0c80695f7b401ae167d52dd977969baf135043bed2a28 Nov 03 07:26:10.139003 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:10.138988 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:26:10.542883 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:10.542797 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" event={"ID":"9b9595de-7246-4ec1-8b88-b2286d081ebb","Type":"ContainerStarted","Data":"9532c2a7e9c4ca27deb0c80695f7b401ae167d52dd977969baf135043bed2a28"} Nov 03 07:26:12.838035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:12.837980 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:13.035661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:13.035616 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:14.132701 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:14.132668 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:26:14.132983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:14.132755 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:26:14.595799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:14.595761 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" event={"ID":"9b9595de-7246-4ec1-8b88-b2286d081ebb","Type":"ContainerStarted","Data":"eb87aa081b8348d6045893cdaef5a587fdf2b3e41ff823fbd47b1bd504746393"} Nov 03 07:26:14.595996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:14.595870 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:14.823704 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:14.823641 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" podStartSLOduration=1.831452351 podStartE2EDuration="5.823622324s" totalImagesPullingTime="3.992169973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:14.771615223 +0000 UTC m=+1634.705143890" watchObservedRunningTime="2026-11-03 07:26:14.823622324 +0000 UTC m=+1634.757150985" Nov 03 07:26:15.607615 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:15.607581 4925 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:26:15.608098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:15.607644 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" podUID="9b9595de-7246-4ec1-8b88-b2286d081ebb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:26:17.946491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:17.946442 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:18.606267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:18.606238 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:26:22.654750 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:22.654706 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:22.909151 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:22.906579 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:23.107471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:23.107436 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:23.165532 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:23.165454 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:26:23.309792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:23.309160 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:23.511929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:23.511835 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:23.711638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:23.711607 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:26.823505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:26.823451 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:26:27.125763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:27.125550 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:26:28.379252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:28.379213 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:28.629975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:28.629896 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:28.867148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:28.867089 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:29.063765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:29.063729 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:29.260410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:29.260356 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:32.852852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:32.852807 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:33.525203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:33.525160 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:33.798093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:33.791555 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:33.996667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:33.996624 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:34.236584 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:34.235356 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:35.239834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:35.239708 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:39.140108 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:26:39.140074 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:26:39.612176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:39.612140 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:39.835412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:39.835369 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:40.557963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:40.557874 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:42.854022 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:42.853968 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:43.060956 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:43.060914 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:45.755632 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:45.755590 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:50.757925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:50.757888 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:26:55.747916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:26:55.747880 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:00.121975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:00.121938 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:00.377349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:00.377265 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:00.942281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:00.942242 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:02.272470 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:02.272439 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:03.020628 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:03.020438 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:03.319553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:03.319500 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:06.220004 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:06.219967 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:07.384560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:07.384526 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:07.592771 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:07.592732 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:07.869391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:07.869359 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:09.139771 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:27:09.139734 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:27:09.386356 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:09.386320 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:11.765548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:11.765501 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:12.702056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:12.702001 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:12.912382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:12.912342 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:15.665917 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:15.665864 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:17.297964 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:17.297928 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:20.353641 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:20.353611 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:27:22.595176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:22.595142 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:22.809202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:22.809161 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:23.012122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:23.011978 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:28.351271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:28.351235 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:29.908868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:29.908830 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:30.160541 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:30.160451 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:31.038799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:31.038761 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:34.046958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:34.046913 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:39.144170 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:27:39.144128 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:27:39.284614 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:39.284578 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:39.515974 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:39.515873 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:39.742805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:39.742766 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:44.922058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:44.921984 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:49.048431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:49.048358 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:49.309982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:49.309889 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:50.056688 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:50.056637 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:51.573079 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:51.573044 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:51.823296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:51.823205 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:55.078616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:55.078577 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:27:57.526946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:27:57.526907 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:00.128761 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:00.128725 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:00.457730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:00.457628 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:04.634647 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:04.634599 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:05.168671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:05.168626 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:07.547534 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:07.547484 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:09.139489 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:09.139451 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:28:10.094961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:10.094923 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:10.351278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:10.351156 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:10.545648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:10.545593 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:15.267083 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:15.267041 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:20.283789 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:20.283745 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:24.669049 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:24.668994 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:26.315962 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:26.315927 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:26.523112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:26.523076 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks"] Nov 03 07:28:26.523370 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:26.523341 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" containerID="cri-o://40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b" gracePeriod=30 Nov 03 07:28:26.574238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:26.574148 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:26.792404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:26.792368 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:27.509492 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:27.509408 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:28.395939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:28.395903 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:29.113066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:29.113025 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:29.675064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:29.674980 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:31.513981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:31.513886 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:32.030262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.030235 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:28:32.051407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.051376 4925 generic.go:356] "Generic (PLEG): container finished" podID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerID="40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b" exitCode=0 Nov 03 07:28:32.051540 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.051430 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" Nov 03 07:28:32.051540 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.051481 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" event={"ID":"4098dcc2-2a51-4035-8c9c-1003fe447dd7","Type":"ContainerDied","Data":"40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b"} Nov 03 07:28:32.051643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.051547 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks" event={"ID":"4098dcc2-2a51-4035-8c9c-1003fe447dd7","Type":"ContainerDied","Data":"a8c5e10f9fe7bbcd628d2365361bf8e8093e110901457e111f53d9f43c56b6e4"} Nov 03 07:28:32.051643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.051579 4925 scope.go:122] "RemoveContainer" containerID="40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b" Nov 03 07:28:32.075098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.075071 4925 scope.go:122] "RemoveContainer" containerID="40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b" Nov 03 07:28:32.075485 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:32.075456 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b\": container with ID starting with 40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b not found: ID does not exist" containerID="40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b" Nov 03 07:28:32.075579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.075503 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b"} err="failed to get container status \"40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b\": rpc error: code = NotFound desc = could not find container \"40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b\": container with ID starting with 40a53bcadac286784bab4ae8f89b73c8f739ba7d829eb39c82220fa6da45b48b not found: ID does not exist" Nov 03 07:28:32.086832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.086807 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-kube-api-access-bnh9n\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-kube-api-access-bnh9n\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.086942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.086867 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-credential-socket\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.086942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.086912 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-token\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.087087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.086973 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-certs\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.087087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.087063 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-envoy\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.087200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.087114 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-socket\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.087200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.087152 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-podinfo\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.090672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.090643 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-token" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:32.091314 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.091291 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-podinfo" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:32.091413 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.091383 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-kube-api-access-bnh9n" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "kube-api-access-bnh9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:32.092618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.092534 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-socket" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.093152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.093125 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-certs" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.094213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.094185 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-envoy" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.095211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.095187 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-credential-socket" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.201145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.198744 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istiod-ca-cert\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.201145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.198855 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-data\") pod \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\" (UID: \"4098dcc2-2a51-4035-8c9c-1003fe447dd7\") " Nov 03 07:28:32.201145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.199326 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istiod-ca-cert" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:32.201145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.199663 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-data" pod "4098dcc2-2a51-4035-8c9c-1003fe447dd7" (UID: "4098dcc2-2a51-4035-8c9c-1003fe447dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204031 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204058 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204073 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204092 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204108 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204123 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204157 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204177 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnh9n\" (UniqueName: \"kubernetes.io/projected/4098dcc2-2a51-4035-8c9c-1003fe447dd7-kube-api-access-bnh9n\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.204657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.204191 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4098dcc2-2a51-4035-8c9c-1003fe447dd7-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.557664 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.557046 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks"] Nov 03 07:28:32.560337 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.560225 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:32.569469 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:32.569439 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks"] Nov 03 07:28:33.151292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:33.151252 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:33.535528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:33.527990 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:33.756435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:33.756399 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:33.968464 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:33.968430 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:34.201791 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:34.201757 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" path="/var/lib/kubelet/pods/4098dcc2-2a51-4035-8c9c-1003fe447dd7/volumes" Nov 03 07:28:34.217047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:34.216996 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:34.433258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:34.433227 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:28:34.458619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:34.458585 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:34.662869 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:34.662835 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:34.885091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:34.885053 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:35.532548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:35.532492 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:35.745168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:35.745111 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:36.506815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:36.506775 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:37.053166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:37.053128 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:37.729046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:37.727434 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:37.965861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:37.965823 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:38.164103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:38.164062 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:38.492093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:38.470758 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:38.717141 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:38.716896 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:39.174446 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:39.174408 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:28:39.541039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:39.540938 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:39.729029 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:39.728977 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:39.947291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:39.947249 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:40.347202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:40.347161 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:40.548523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:40.548485 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:40.762031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:40.761923 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:41.743733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:41.743697 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr"] Nov 03 07:28:41.744176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:41.743961 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" containerID="cri-o://9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0" gracePeriod=30 Nov 03 07:28:42.288465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.288426 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w"] Nov 03 07:28:42.288741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.288711 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" containerID="cri-o://e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42" gracePeriod=30 Nov 03 07:28:42.295975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.295948 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf"] Nov 03 07:28:42.296219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.296194 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" containerID="cri-o://1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530" gracePeriod=30 Nov 03 07:28:42.340505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.340464 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l"] Nov 03 07:28:42.340757 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.340726 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" containerID="cri-o://27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568" gracePeriod=30 Nov 03 07:28:42.346221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.346187 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb"] Nov 03 07:28:42.346448 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:42.346426 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" containerID="cri-o://2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc" gracePeriod=30 Nov 03 07:28:43.687310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:43.687263 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:44.007405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:44.007306 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:44.260746 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:44.260644 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:44.490521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:44.490487 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:45.093703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:45.093667 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:47.054647 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.054623 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:28:47.123522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.123483 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-socket\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.123707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.123611 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-token\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-token\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.123707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.123646 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istiod-ca-cert\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.123824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.123753 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-socket" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.123824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.123805 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-envoy\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.123924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.123886 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-credential-socket\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.124072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.124055 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-podinfo\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.124135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.124104 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istiod-ca-cert" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.124198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.124133 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-certs\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.124198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.124169 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-data\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.124295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.124215 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-kube-api-access-hssdl\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-kube-api-access-hssdl\") pod \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\" (UID: \"2166463c-54cc-4a4d-831d-2a4a72a3dacb\") " Nov 03 07:28:47.124605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.124583 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-certs" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.124928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.124890 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-data" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.125211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.125177 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-credential-socket" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.125211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.125190 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.125373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.125213 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.125373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.125228 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.125373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.125242 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.126638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.126613 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-token" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.126762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.126739 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-envoy" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.127212 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.127189 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-kube-api-access-hssdl" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "kube-api-access-hssdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.127322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.127304 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-podinfo" pod "2166463c-54cc-4a4d-831d-2a4a72a3dacb" (UID: "2166463c-54cc-4a4d-831d-2a4a72a3dacb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.226889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.226856 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.226889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.226884 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.226889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.226894 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2166463c-54cc-4a4d-831d-2a4a72a3dacb-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.227147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.226903 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2166463c-54cc-4a4d-831d-2a4a72a3dacb-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.227147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.226912 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hssdl\" (UniqueName: \"kubernetes.io/projected/2166463c-54cc-4a4d-831d-2a4a72a3dacb-kube-api-access-hssdl\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.278606 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.278513 4925 generic.go:356] "Generic (PLEG): container finished" podID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerID="9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0" exitCode=0 Nov 03 07:28:47.278881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.278607 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" event={"ID":"2166463c-54cc-4a4d-831d-2a4a72a3dacb","Type":"ContainerDied","Data":"9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0"} Nov 03 07:28:47.278881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.278660 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" event={"ID":"2166463c-54cc-4a4d-831d-2a4a72a3dacb","Type":"ContainerDied","Data":"6c8c7487d4f8988c16bd3c8464f18de33973f00defece6adb115e95c040dc996"} Nov 03 07:28:47.278881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.278689 4925 scope.go:122] "RemoveContainer" containerID="9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0" Nov 03 07:28:47.279210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.279191 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr" Nov 03 07:28:47.305857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.305827 4925 scope.go:122] "RemoveContainer" containerID="9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0" Nov 03 07:28:47.306259 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:47.306228 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0\": container with ID starting with 9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0 not found: ID does not exist" containerID="9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0" Nov 03 07:28:47.306368 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.306273 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0"} err="failed to get container status \"9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0\": rpc error: code = NotFound desc = could not find container \"9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0\": container with ID starting with 9189607457ece8a9247f03ded4b22ffd00891fb6ced75e831d322ac15ddec4b0 not found: ID does not exist" Nov 03 07:28:47.463501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.463472 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr"] Nov 03 07:28:47.468309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.468286 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr"] Nov 03 07:28:47.670072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.670040 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:28:47.746669 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.746626 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-envoy\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.746933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.746766 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-podinfo\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.746933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.746799 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-data\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.746933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.746834 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-token\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-token\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.746933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.746865 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-credential-socket\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.747222 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.746934 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-socket\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.747222 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.747101 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-kube-api-access-nktt6\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-kube-api-access-nktt6\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.747375 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.747351 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-certs\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.747414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.747392 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98ba875-0d62-41da-90d9-b8a8b8724e47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98ba875-0d62-41da-90d9-b8a8b8724e47-istiod-ca-cert\") pod \"c98ba875-0d62-41da-90d9-b8a8b8724e47\" (UID: \"c98ba875-0d62-41da-90d9-b8a8b8724e47\") " Nov 03 07:28:47.748131 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.747903 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-credential-socket" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.748282 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.748257 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98ba875-0d62-41da-90d9-b8a8b8724e47-istiod-ca-cert" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.748512 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.748498 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-socket" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.749309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.749281 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.749417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.749309 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-data" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.749552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.749526 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-certs" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.751606 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.751582 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-kube-api-access-nktt6" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "kube-api-access-nktt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.752105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.752062 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-podinfo" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.752285 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.752119 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-envoy" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.753576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.753556 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-token" pod "c98ba875-0d62-41da-90d9-b8a8b8724e47" (UID: "c98ba875-0d62-41da-90d9-b8a8b8724e47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.760631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.760611 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:28:47.763753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.763735 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:28:47.822580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.822521 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:47.854668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854635 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-token\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-token\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.854832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854681 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istiod-ca-cert\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.854832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854735 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-podinfo\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.854832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854771 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-socket\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.854832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854826 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-data\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.855057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854853 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-credential-socket\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.855057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854893 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-kube-api-access-pmlvk\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-kube-api-access-pmlvk\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.855057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854947 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-envoy\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.855057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.854978 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-socket\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.855057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855043 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-credential-socket\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.855242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855178 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-certs\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.855242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855212 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-kube-api-access-tjbsr\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-kube-api-access-tjbsr\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.855302 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855266 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-podinfo\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.855337 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855326 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86af9085-3448-4f1f-85fa-0d3585e307fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86af9085-3448-4f1f-85fa-0d3585e307fe-istiod-ca-cert\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.855399 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855380 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-data\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.855454 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855440 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-token\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-token\") pod \"86af9085-3448-4f1f-85fa-0d3585e307fe\" (UID: \"86af9085-3448-4f1f-85fa-0d3585e307fe\") " Nov 03 07:28:47.856041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855551 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-certs\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.856041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855640 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-envoy\") pod \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\" (UID: \"5524a86d-7cf2-4894-bdb7-6701d7f8a681\") " Nov 03 07:28:47.856041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855753 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-credential-socket" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.856041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855863 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-data" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.856041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855888 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-socket" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.856041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855911 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istiod-ca-cert" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.856041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.855985 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-credential-socket" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.856349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856154 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-data" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.856389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856349 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-certs" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.856621 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856597 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-socket" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.856621 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856616 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856630 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856640 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856649 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98ba875-0d62-41da-90d9-b8a8b8724e47-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856657 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856665 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856674 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856681 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856702 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856710 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856722 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856748 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856758 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98ba875-0d62-41da-90d9-b8a8b8724e47-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856766 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.856790 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856775 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nktt6\" (UniqueName: \"kubernetes.io/projected/c98ba875-0d62-41da-90d9-b8a8b8724e47-kube-api-access-nktt6\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.857517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.856805 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86af9085-3448-4f1f-85fa-0d3585e307fe-istiod-ca-cert" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.857517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.857057 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-certs" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.864898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.860213 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-token" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.864898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.860215 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-podinfo" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.864898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.860571 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-envoy" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.864898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.860808 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-podinfo" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.864898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.861362 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-kube-api-access-tjbsr" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "kube-api-access-tjbsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.882240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.874800 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:28:47.895006 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.894256 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-token" pod "86af9085-3448-4f1f-85fa-0d3585e307fe" (UID: "86af9085-3448-4f1f-85fa-0d3585e307fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.895519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.895472 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-kube-api-access-pmlvk" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "kube-api-access-pmlvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.898468 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.898443 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-envoy" pod "5524a86d-7cf2-4894-bdb7-6701d7f8a681" (UID: "5524a86d-7cf2-4894-bdb7-6701d7f8a681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.960692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.960612 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-podinfo\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961098 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-certs\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961302 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961278 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-credential-socket\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961344 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-data\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961413 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-certs" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.961518 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961501 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2c4453-3826-478d-987f-9f9d817dcd25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c4453-3826-478d-987f-9f9d817dcd25-istiod-ca-cert\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961618 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-socket\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961620 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-credential-socket" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.961741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961688 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-kube-api-access-slkds\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-kube-api-access-slkds\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961728 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-envoy\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961799 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-data" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.961840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961804 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-token\") pod \"bb2c4453-3826-478d-987f-9f9d817dcd25\" (UID: \"bb2c4453-3826-478d-987f-9f9d817dcd25\") " Nov 03 07:28:47.961973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.961954 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-socket" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.962473 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962449 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2c4453-3826-478d-987f-9f9d817dcd25-istiod-ca-cert" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962470 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962493 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962508 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjbsr\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-kube-api-access-tjbsr\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962525 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962540 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86af9085-3448-4f1f-85fa-0d3585e307fe-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962554 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962567 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962580 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962593 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962608 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962621 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962634 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962648 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlvk\" (UniqueName: \"kubernetes.io/projected/5524a86d-7cf2-4894-bdb7-6701d7f8a681-kube-api-access-pmlvk\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962661 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.962981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.962674 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86af9085-3448-4f1f-85fa-0d3585e307fe-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.965361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.965338 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-kube-api-access-slkds" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "kube-api-access-slkds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.967676 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.967652 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-envoy" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.967676 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.967662 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-token" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.969948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:47.969923 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-podinfo" pod "bb2c4453-3826-478d-987f-9f9d817dcd25" (UID: "bb2c4453-3826-478d-987f-9f9d817dcd25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:48.029389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.029338 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:48.066573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.066532 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkds\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-kube-api-access-slkds\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.066573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.066570 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.067183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.066585 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.067183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.066599 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c4453-3826-478d-987f-9f9d817dcd25-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.067183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.066613 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c4453-3826-478d-987f-9f9d817dcd25-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.197849 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.197752 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" path="/var/lib/kubelet/pods/2166463c-54cc-4a4d-831d-2a4a72a3dacb/volumes" Nov 03 07:28:48.224117 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.224084 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:48.290822 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.290788 4925 generic.go:356] "Generic (PLEG): container finished" podID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerID="1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530" exitCode=0 Nov 03 07:28:48.291028 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.290869 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" event={"ID":"86af9085-3448-4f1f-85fa-0d3585e307fe","Type":"ContainerDied","Data":"1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530"} Nov 03 07:28:48.291028 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.290879 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" Nov 03 07:28:48.291028 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.290934 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf" event={"ID":"86af9085-3448-4f1f-85fa-0d3585e307fe","Type":"ContainerDied","Data":"bd73d72c7c90f9814f0a00836bb6010037efb58539464a8f207d26e68eb2ec42"} Nov 03 07:28:48.291028 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.290970 4925 scope.go:122] "RemoveContainer" containerID="1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530" Nov 03 07:28:48.292912 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.292877 4925 generic.go:356] "Generic (PLEG): container finished" podID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerID="27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568" exitCode=0 Nov 03 07:28:48.293062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.292984 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" event={"ID":"c98ba875-0d62-41da-90d9-b8a8b8724e47","Type":"ContainerDied","Data":"27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568"} Nov 03 07:28:48.293135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.293066 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" event={"ID":"c98ba875-0d62-41da-90d9-b8a8b8724e47","Type":"ContainerDied","Data":"da3114aa42a70aba497af3b220c2628b597e1872fa424c559d32e66d43244e37"} Nov 03 07:28:48.293335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.293317 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l" Nov 03 07:28:48.296468 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.296445 4925 generic.go:356] "Generic (PLEG): container finished" podID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerID="e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42" exitCode=0 Nov 03 07:28:48.296583 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.296482 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" event={"ID":"5524a86d-7cf2-4894-bdb7-6701d7f8a681","Type":"ContainerDied","Data":"e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42"} Nov 03 07:28:48.296583 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.296516 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" Nov 03 07:28:48.296583 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.296534 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w" event={"ID":"5524a86d-7cf2-4894-bdb7-6701d7f8a681","Type":"ContainerDied","Data":"dd37f0a431eabd1cc2182c0aa52d2753ade8844fc839e6e24733e66570ef3f81"} Nov 03 07:28:48.298210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.298191 4925 generic.go:356] "Generic (PLEG): container finished" podID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerID="2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc" exitCode=0 Nov 03 07:28:48.298298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.298237 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" Nov 03 07:28:48.298349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.298243 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" event={"ID":"bb2c4453-3826-478d-987f-9f9d817dcd25","Type":"ContainerDied","Data":"2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc"} Nov 03 07:28:48.298349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.298338 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb" event={"ID":"bb2c4453-3826-478d-987f-9f9d817dcd25","Type":"ContainerDied","Data":"6f7bc6c2a86eeba8ca925888e5871572c1866a0365e2f79d21a2eb099d118578"} Nov 03 07:28:48.312339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.308302 4925 scope.go:122] "RemoveContainer" containerID="1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530" Nov 03 07:28:48.316517 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:48.314979 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530\": container with ID starting with 1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530 not found: ID does not exist" containerID="1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530" Nov 03 07:28:48.316517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.315107 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530"} err="failed to get container status \"1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530\": rpc error: code = NotFound desc = could not find container \"1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530\": container with ID starting with 1095564f0ea6cd826200b3e66d7ce8e90ec0dede3a40c821d3b564ebb35a6530 not found: ID does not exist" Nov 03 07:28:48.316517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.315144 4925 scope.go:122] "RemoveContainer" containerID="27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568" Nov 03 07:28:48.327978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.327958 4925 scope.go:122] "RemoveContainer" containerID="27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568" Nov 03 07:28:48.328355 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:48.328334 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568\": container with ID starting with 27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568 not found: ID does not exist" containerID="27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568" Nov 03 07:28:48.328425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.328365 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568"} err="failed to get container status \"27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568\": rpc error: code = NotFound desc = could not find container \"27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568\": container with ID starting with 27f5227564844bf22f6ae2c4a7a16861d1929874be79a2e6adbb186b63ad9568 not found: ID does not exist" Nov 03 07:28:48.328425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.328384 4925 scope.go:122] "RemoveContainer" containerID="e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42" Nov 03 07:28:48.337788 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.337768 4925 scope.go:122] "RemoveContainer" containerID="e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42" Nov 03 07:28:48.338074 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:48.338057 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42\": container with ID starting with e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42 not found: ID does not exist" containerID="e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42" Nov 03 07:28:48.338127 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.338081 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42"} err="failed to get container status \"e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42\": rpc error: code = NotFound desc = could not find container \"e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42\": container with ID starting with e55e29b6132106ea20d0d5135fc404a0f2b3a98156955d96f4e7d982e78c6a42 not found: ID does not exist" Nov 03 07:28:48.338127 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.338097 4925 scope.go:122] "RemoveContainer" containerID="2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc" Nov 03 07:28:48.347522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.347500 4925 scope.go:122] "RemoveContainer" containerID="2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc" Nov 03 07:28:48.347822 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:28:48.347801 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc\": container with ID starting with 2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc not found: ID does not exist" containerID="2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc" Nov 03 07:28:48.347901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.347830 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc"} err="failed to get container status \"2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc\": rpc error: code = NotFound desc = could not find container \"2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc\": container with ID starting with 2a6be15437a8974c9cbf6a52d4d45c43bac37b5887ec4439487730055a806dfc not found: ID does not exist" Nov 03 07:28:48.468386 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.468299 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l"] Nov 03 07:28:48.476075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.476047 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l"] Nov 03 07:28:48.598946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.598900 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf"] Nov 03 07:28:48.605428 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.605398 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf"] Nov 03 07:28:48.718991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.718912 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb"] Nov 03 07:28:48.728429 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.728400 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb"] Nov 03 07:28:48.785671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.785638 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:48.848208 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.848172 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w"] Nov 03 07:28:48.854435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:48.854403 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w"] Nov 03 07:28:49.005619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:49.005503 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:49.604432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:49.604396 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:50.191919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:50.191889 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" path="/var/lib/kubelet/pods/5524a86d-7cf2-4894-bdb7-6701d7f8a681/volumes" Nov 03 07:28:50.192471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:50.192456 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" path="/var/lib/kubelet/pods/86af9085-3448-4f1f-85fa-0d3585e307fe/volumes" Nov 03 07:28:50.192912 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:50.192899 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" path="/var/lib/kubelet/pods/bb2c4453-3826-478d-987f-9f9d817dcd25/volumes" Nov 03 07:28:50.193668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:50.193652 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" path="/var/lib/kubelet/pods/c98ba875-0d62-41da-90d9-b8a8b8724e47/volumes" Nov 03 07:28:50.582366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:50.582324 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:51.366233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:51.366192 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:51.685822 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:51.685727 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:56.178140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:56.176105 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:56.451267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:56.451178 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:56.649259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:56.649217 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:57.022768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:57.022737 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:28:57.231039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:28:57.230930 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:00.808763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:00.808721 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:01.646569 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:01.646537 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:29:01.647061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:01.647036 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:29:01.650881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:01.650862 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:29:01.651405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:01.651384 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:29:01.804350 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:01.804253 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:06.801580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:06.801544 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:09.139876 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:09.139839 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:29:10.255539 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:10.255505 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:12.557974 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:12.557932 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:16.292777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:16.292738 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:16.930323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:16.930271 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:20.615413 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:20.615373 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:22.395358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:22.395263 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:24.266678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:24.266641 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:27.023231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:27.023196 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl"] Nov 03 07:29:27.023691 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:27.023465 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" containerID="cri-o://86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e" gracePeriod=30 Nov 03 07:29:27.094795 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:27.094762 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7"] Nov 03 07:29:27.095565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:27.095532 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" containerID="cri-o://7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c" gracePeriod=30 Nov 03 07:29:27.100214 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:27.100192 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl"] Nov 03 07:29:27.100416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:27.100390 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" containerID="cri-o://83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5" gracePeriod=30 Nov 03 07:29:27.797689 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:27.797649 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:28.194459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:28.194419 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:28.393493 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:28.393456 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:29.600172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:29.600140 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:29:29.881249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:29.881160 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:30.078560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:30.078526 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:30.281237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:30.281161 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:32.434414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.434386 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:29:32.436820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.436797 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:29:32.483562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.483537 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:29:32.577862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.577787 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-token\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-token\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.577862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.577821 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-credential-socket\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.577862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.577850 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-certs\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.578180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.577885 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-certs\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.578180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.577916 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-envoy\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.578180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.577932 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-data\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.578180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.577974 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-credential-socket\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.578180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578083 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-data\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.578180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578114 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-socket\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.578180 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578139 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-credential-socket" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578150 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istiod-ca-cert\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578193 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-certs" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578231 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-kube-api-access-wrs5s\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-kube-api-access-wrs5s\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578290 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d907b61f-11a2-41fe-98b9-da8d93cc952d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d907b61f-11a2-41fe-98b9-da8d93cc952d-istiod-ca-cert\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578300 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-data" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578311 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-certs" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578345 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-credential-socket\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578377 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-data" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578374 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-credential-socket" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578428 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-envoy\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578468 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-token\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-token\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.578526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578523 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-kube-api-access-z24dm\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-kube-api-access-z24dm\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578559 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-certs\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578588 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-socket\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578559 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-credential-socket" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578592 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istiod-ca-cert" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578524 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-socket" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578641 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-podinfo\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578679 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-podinfo\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578730 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-podinfo\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578783 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-socket\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578816 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istiod-ca-cert\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578827 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-certs" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578841 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-envoy\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578882 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-token\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-token\") pod \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\" (UID: \"bf603287-bda9-4061-bae6-a1dd53ec6dc4\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578887 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d907b61f-11a2-41fe-98b9-da8d93cc952d-istiod-ca-cert" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578913 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-kube-api-access-hb8rk\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-kube-api-access-hb8rk\") pod \"d907b61f-11a2-41fe-98b9-da8d93cc952d\" (UID: \"d907b61f-11a2-41fe-98b9-da8d93cc952d\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.578993 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-data\") pod \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\" (UID: \"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1\") " Nov 03 07:29:32.579335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579061 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-socket" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579833 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d907b61f-11a2-41fe-98b9-da8d93cc952d-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579858 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579873 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579886 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579900 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579913 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579920 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istiod-ca-cert" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579928 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.579963 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.580040 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.580060 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.580075 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.580090 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.580240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.580182 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-data" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.580975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.580423 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-socket" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.581360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.581331 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-token" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:32.581360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.581343 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-kube-api-access-z24dm" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "kube-api-access-z24dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:32.581532 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.581469 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-envoy" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.582076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.582051 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-podinfo" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:29:32.582330 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.582304 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-kube-api-access-wrs5s" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "kube-api-access-wrs5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:32.583042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.582951 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-envoy" pod "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" (UID: "4d39724d-ee42-4c1a-94c6-5bb6349ca3e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.583568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.583540 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-kube-api-access-hb8rk" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "kube-api-access-hb8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:32.583568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.583552 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-envoy" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:32.583568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.583545 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-token" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:32.583766 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.583649 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-token" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:32.583821 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.583806 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-podinfo" pod "bf603287-bda9-4061-bae6-a1dd53ec6dc4" (UID: "bf603287-bda9-4061-bae6-a1dd53ec6dc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:29:32.583946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.583933 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-podinfo" pod "d907b61f-11a2-41fe-98b9-da8d93cc952d" (UID: "d907b61f-11a2-41fe-98b9-da8d93cc952d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:29:32.682045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.681995 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682257 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682237 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682263 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682280 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrs5s\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-kube-api-access-wrs5s\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682299 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682314 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682328 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24dm\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-kube-api-access-z24dm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682341 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682354 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682368 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682382 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682396 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682409 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d907b61f-11a2-41fe-98b9-da8d93cc952d-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682423 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf603287-bda9-4061-bae6-a1dd53ec6dc4-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.682680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.682437 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb8rk\" (UniqueName: \"kubernetes.io/projected/d907b61f-11a2-41fe-98b9-da8d93cc952d-kube-api-access-hb8rk\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:32.686953 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.686927 4925 generic.go:356] "Generic (PLEG): container finished" podID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerID="83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5" exitCode=0 Nov 03 07:29:32.687104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.686990 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" Nov 03 07:29:32.687104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.687064 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" event={"ID":"bf603287-bda9-4061-bae6-a1dd53ec6dc4","Type":"ContainerDied","Data":"83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5"} Nov 03 07:29:32.687196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.687166 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl" event={"ID":"bf603287-bda9-4061-bae6-a1dd53ec6dc4","Type":"ContainerDied","Data":"8f8445c1ee61063ec3b1a0031b941ce9329230f66689e4dc08b80e4f14180220"} Nov 03 07:29:32.687238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.687202 4925 scope.go:122] "RemoveContainer" containerID="83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5" Nov 03 07:29:32.688873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.688844 4925 generic.go:356] "Generic (PLEG): container finished" podID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerID="86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e" exitCode=0 Nov 03 07:29:32.688968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.688880 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" event={"ID":"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1","Type":"ContainerDied","Data":"86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e"} Nov 03 07:29:32.688968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.688913 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" Nov 03 07:29:32.688968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.688922 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl" event={"ID":"4d39724d-ee42-4c1a-94c6-5bb6349ca3e1","Type":"ContainerDied","Data":"d5ec6bd51048e531fb0499ea965f7e8183854fb9f4e5e3be2b6af668571e72d4"} Nov 03 07:29:32.690795 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.690775 4925 generic.go:356] "Generic (PLEG): container finished" podID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerID="7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c" exitCode=0 Nov 03 07:29:32.690902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.690824 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" event={"ID":"d907b61f-11a2-41fe-98b9-da8d93cc952d","Type":"ContainerDied","Data":"7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c"} Nov 03 07:29:32.690902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.690851 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" Nov 03 07:29:32.690902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.690873 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7" event={"ID":"d907b61f-11a2-41fe-98b9-da8d93cc952d","Type":"ContainerDied","Data":"1aea9391c2d9e5f2b14f89acad9cf1455296fb27b19ab0c59cd9000430b5641d"} Nov 03 07:29:32.699098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.699078 4925 scope.go:122] "RemoveContainer" containerID="83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5" Nov 03 07:29:32.699396 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:32.699376 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5\": container with ID starting with 83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5 not found: ID does not exist" containerID="83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5" Nov 03 07:29:32.699472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.699403 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5"} err="failed to get container status \"83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5\": rpc error: code = NotFound desc = could not find container \"83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5\": container with ID starting with 83ec207a5770a80c824fcba9551688123e42a1e6160283b267b36c7fc78f24c5 not found: ID does not exist" Nov 03 07:29:32.699472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.699418 4925 scope.go:122] "RemoveContainer" containerID="86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e" Nov 03 07:29:32.708873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.708848 4925 scope.go:122] "RemoveContainer" containerID="86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e" Nov 03 07:29:32.709126 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:32.709108 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e\": container with ID starting with 86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e not found: ID does not exist" containerID="86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e" Nov 03 07:29:32.709186 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.709131 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e"} err="failed to get container status \"86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e\": rpc error: code = NotFound desc = could not find container \"86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e\": container with ID starting with 86e4f09b0f9ca8def1993452463b5a34a668b7afefa80a56c7304ab21f12c74e not found: ID does not exist" Nov 03 07:29:32.709186 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.709147 4925 scope.go:122] "RemoveContainer" containerID="7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c" Nov 03 07:29:32.717991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.717975 4925 scope.go:122] "RemoveContainer" containerID="7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c" Nov 03 07:29:32.718275 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:32.718259 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c\": container with ID starting with 7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c not found: ID does not exist" containerID="7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c" Nov 03 07:29:32.718327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.718280 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c"} err="failed to get container status \"7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c\": rpc error: code = NotFound desc = could not find container \"7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c\": container with ID starting with 7d295d93ac20192151a795a8293aced42a5feb934edf6b5ca35006cd4276bb3c not found: ID does not exist" Nov 03 07:29:32.892983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.892894 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:32.997711 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:32.997675 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl"] Nov 03 07:29:33.016270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:33.014924 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl"] Nov 03 07:29:33.106168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:33.106138 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:33.132779 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:33.132749 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl"] Nov 03 07:29:33.143144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:33.143082 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl"] Nov 03 07:29:33.256863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:33.256833 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7"] Nov 03 07:29:33.263717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:33.263691 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7"] Nov 03 07:29:33.306657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:33.306624 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:34.195455 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:34.195426 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" path="/var/lib/kubelet/pods/4d39724d-ee42-4c1a-94c6-5bb6349ca3e1/volumes" Nov 03 07:29:34.195974 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:34.195954 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" path="/var/lib/kubelet/pods/bf603287-bda9-4061-bae6-a1dd53ec6dc4/volumes" Nov 03 07:29:34.196471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:34.196456 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" path="/var/lib/kubelet/pods/d907b61f-11a2-41fe-98b9-da8d93cc952d/volumes" Nov 03 07:29:38.037329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:38.037290 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:39.139621 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:39.139588 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:29:42.096791 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:42.096710 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl"] Nov 03 07:29:42.097247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:42.096961 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" containerID="cri-o://cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce" gracePeriod=30 Nov 03 07:29:42.970144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:42.970002 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:43.164999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:43.164965 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:43.359666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:43.359633 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:47.370651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.370574 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:29:47.492087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.492052 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-podinfo\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.492579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.492555 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-certs\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.492860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.492794 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-data\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.492860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.492801 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-certs" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:47.492860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.492842 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f28e7d-5afd-4721-9700-d2e7163486cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f28e7d-5afd-4721-9700-d2e7163486cd-istiod-ca-cert\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.493126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.492880 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-credential-socket\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.493126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.492984 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-envoy\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.493126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.493070 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-socket\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.493294 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.493124 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-data" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:47.493294 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.493215 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-kube-api-access-t9l8v\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-kube-api-access-t9l8v\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.493404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.493309 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-token\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-token\") pod \"44f28e7d-5afd-4721-9700-d2e7163486cd\" (UID: \"44f28e7d-5afd-4721-9700-d2e7163486cd\") " Nov 03 07:29:47.493404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.493313 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-credential-socket" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:47.494118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.493603 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f28e7d-5afd-4721-9700-d2e7163486cd-istiod-ca-cert" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:29:47.494118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.493993 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.494118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.494033 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.494118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.494047 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f28e7d-5afd-4721-9700-d2e7163486cd-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.494118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.494060 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.494118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.494090 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-socket" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:47.496125 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.495990 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-podinfo" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:29:47.496514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.496485 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-envoy" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:29:47.496592 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.496549 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-kube-api-access-t9l8v" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "kube-api-access-t9l8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:47.496592 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.496568 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-token" pod "44f28e7d-5afd-4721-9700-d2e7163486cd" (UID: "44f28e7d-5afd-4721-9700-d2e7163486cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:47.595385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.595345 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9l8v\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-kube-api-access-t9l8v\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.595385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.595383 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.595568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.595398 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.595568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.595412 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.595568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.595425 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f28e7d-5afd-4721-9700-d2e7163486cd-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:47.612135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.612094 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:47.839768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.838049 4925 generic.go:356] "Generic (PLEG): container finished" podID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerID="cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce" exitCode=0 Nov 03 07:29:47.839768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.838258 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" event={"ID":"44f28e7d-5afd-4721-9700-d2e7163486cd","Type":"ContainerDied","Data":"cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce"} Nov 03 07:29:47.839768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.838311 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" event={"ID":"44f28e7d-5afd-4721-9700-d2e7163486cd","Type":"ContainerDied","Data":"a934683e7922fd314184accafac5d1574b421cfa83ebdbf4c4a6777517529cf5"} Nov 03 07:29:47.839768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.838340 4925 scope.go:122] "RemoveContainer" containerID="cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce" Nov 03 07:29:47.839768 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.838518 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl" Nov 03 07:29:47.905036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.886337 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:47.916233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.916207 4925 scope.go:122] "RemoveContainer" containerID="cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce" Nov 03 07:29:47.916683 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:47.916650 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce\": container with ID starting with cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce not found: ID does not exist" containerID="cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce" Nov 03 07:29:47.916778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:47.916695 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce"} err="failed to get container status \"cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce\": rpc error: code = NotFound desc = could not find container \"cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce\": container with ID starting with cb025bd4749fa94b7a6c409de2a2cce241b803b07ed17f1257aefed79ad5d0ce not found: ID does not exist" Nov 03 07:29:48.024802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:48.024764 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl"] Nov 03 07:29:48.084293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:48.083529 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl"] Nov 03 07:29:48.193901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:48.193818 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" path="/var/lib/kubelet/pods/44f28e7d-5afd-4721-9700-d2e7163486cd/volumes" Nov 03 07:29:49.139520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:49.139487 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:49.377862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:49.377826 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:49.570880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:49.570845 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:50.087725 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:50.087681 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:54.511214 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.511174 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228"] Nov 03 07:29:54.511939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.511919 4925 cpu_manager.go:414] "containerMap: removing container" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" Nov 03 07:29:54.512050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.511944 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" Nov 03 07:29:54.512050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.511986 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" Nov 03 07:29:54.512050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.511994 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" Nov 03 07:29:54.512050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512026 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" Nov 03 07:29:54.512050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512034 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" Nov 03 07:29:54.512050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512049 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512058 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512067 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512075 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512106 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512114 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512133 4925 cpu_manager.go:414] "containerMap: removing container" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512141 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512159 4925 cpu_manager.go:414] "containerMap: removing container" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512165 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512178 4925 cpu_manager.go:414] "containerMap: removing container" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512184 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512194 4925 cpu_manager.go:414] "containerMap: removing container" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" Nov 03 07:29:54.512329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512201 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512343 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d39724d-ee42-4c1a-94c6-5bb6349ca3e1" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512355 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4098dcc2-2a51-4035-8c9c-1003fe447dd7" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512361 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2166463c-54cc-4a4d-831d-2a4a72a3dacb" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512368 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5524a86d-7cf2-4894-bdb7-6701d7f8a681" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512375 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d907b61f-11a2-41fe-98b9-da8d93cc952d" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512383 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2c4453-3826-478d-987f-9f9d817dcd25" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512389 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98ba875-0d62-41da-90d9-b8a8b8724e47" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512395 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="86af9085-3448-4f1f-85fa-0d3585e307fe" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512401 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf603287-bda9-4061-bae6-a1dd53ec6dc4" containerName="istio-proxy" Nov 03 07:29:54.512677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.512406 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f28e7d-5afd-4721-9700-d2e7163486cd" containerName="istio-proxy" Nov 03 07:29:54.515773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.515752 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.534552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.534515 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.534652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.534576 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjg9l\" (UniqueName: \"kubernetes.io/projected/7df9991f-9961-49f6-9e5d-253d7f83482e-kube-api-access-hjg9l\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.534720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.534698 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.534909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.534896 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df9991f-9961-49f6-9e5d-253d7f83482e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.571856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.571829 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-09fd61-serving-cert\"" Nov 03 07:29:54.581613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.581573 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228"] Nov 03 07:29:54.636777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.636742 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df9991f-9961-49f6-9e5d-253d7f83482e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.636952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.636818 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.636952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.636845 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjg9l\" (UniqueName: \"kubernetes.io/projected/7df9991f-9961-49f6-9e5d-253d7f83482e-kube-api-access-hjg9l\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.636952 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:54.636936 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-serving-cert: secret "maas-api-e2e-shared-a-w2-09fd61-serving-cert" not found Nov 03 07:29:54.637184 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:29:54.637044 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls podName:7df9991f-9961-49f6-9e5d-253d7f83482e nodeName:}" failed. No retries permitted until 2026-11-03 07:29:55.137023085 +0000 UTC m=+1855.070551733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls") pod "maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" (UID: "7df9991f-9961-49f6-9e5d-253d7f83482e") : secret "maas-api-e2e-shared-a-w2-09fd61-serving-cert" not found Nov 03 07:29:54.637184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.637076 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.637555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.637529 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df9991f-9961-49f6-9e5d-253d7f83482e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.640172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.639838 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.703982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.703945 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjg9l\" (UniqueName: \"kubernetes.io/projected/7df9991f-9961-49f6-9e5d-253d7f83482e-kube-api-access-hjg9l\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:54.764194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:54.764091 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:55.003694 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:55.003651 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:55.169987 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:55.169958 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:55.172838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:55.172815 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:55.433740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:55.433650 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:55.966391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:55.966350 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228"] Nov 03 07:29:55.971513 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:29:55.971481 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df9991f_9961_49f6_9e5d_253d7f83482e.slice/crio-e02743125fb204241a86ae8c431508710265a782de5ca7205211d31b6755551c WatchSource:0}: Error finding container e02743125fb204241a86ae8c431508710265a782de5ca7205211d31b6755551c: Status 404 returned error can't find the container with id e02743125fb204241a86ae8c431508710265a782de5ca7205211d31b6755551c Nov 03 07:29:56.423926 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:56.423891 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:57.098812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:57.098259 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" event={"ID":"7df9991f-9961-49f6-9e5d-253d7f83482e","Type":"ContainerStarted","Data":"e02743125fb204241a86ae8c431508710265a782de5ca7205211d31b6755551c"} Nov 03 07:29:58.119247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.119183 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:58.128039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.121142 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:58.319794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.319756 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" event={"ID":"7df9991f-9961-49f6-9e5d-253d7f83482e","Type":"ContainerStarted","Data":"248eb11b73ad4e4e299b049c0ba1a60b5eed6ca0ba088bb6a698d778ab3bf231"} Nov 03 07:29:58.321205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.321182 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:29:58.449102 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.447088 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:58.479504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.479448 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" podStartSLOduration=3.548564932 podStartE2EDuration="4.479428772s" totalImagesPullingTime="930.86384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:29:58.459991899 +0000 UTC m=+1858.393520556" watchObservedRunningTime="2026-11-03 07:29:58.479428772 +0000 UTC m=+1858.412957447" Nov 03 07:29:58.553900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.553241 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 07:29:58.785487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.785379 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:58.826219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.826172 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:58.990038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:58.987310 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:59.142036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.133327 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:59.142036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.133912 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:29:59.221626 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.220588 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:59.270152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.270113 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:29:59.290178 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.270407 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:59.418462 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.418379 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:29:59.488373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.488215 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:29:59.749241 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.749157 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:29:59.749494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:29:59.749461 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:30:00.207571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.207536 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr"] Nov 03 07:30:00.212656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.212632 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9"] Nov 03 07:30:00.212804 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.212760 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.216586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.216565 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.222117 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.222097 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp"] Nov 03 07:30:00.226161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.226139 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.235929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.235904 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:00.247763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.247732 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n"] Nov 03 07:30:00.251921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.251901 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.271076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.271050 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr"] Nov 03 07:30:00.303226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.303191 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9"] Nov 03 07:30:00.304339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.304311 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4597027-e561-4878-84e1-82aca3789b86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.310286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.310248 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c181f7c-4477-4e80-b0b7-1264f7a28291-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.310658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.310631 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzc7\" (UniqueName: \"kubernetes.io/projected/4dd33f98-33ac-4fa7-b77d-313e19ce5770-kube-api-access-sjzc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.310782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.310691 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd33f98-33ac-4fa7-b77d-313e19ce5770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.310782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.310775 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpl9s\" (UniqueName: \"kubernetes.io/projected/e4597027-e561-4878-84e1-82aca3789b86-kube-api-access-xpl9s\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.310903 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.310808 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c500f-e8bf-485f-b8e2-f3635910d25c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.310903 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.310829 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hls46\" (UniqueName: \"kubernetes.io/projected/2c181f7c-4477-4e80-b0b7-1264f7a28291-kube-api-access-hls46\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.310903 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.310851 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvwk\" (UniqueName: \"kubernetes.io/projected/706c500f-e8bf-485f-b8e2-f3635910d25c-kube-api-access-2rvwk\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.311027 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.307076 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp"] Nov 03 07:30:00.346279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.346245 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:30:00.382839 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.382801 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n"] Nov 03 07:30:00.412477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412423 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c181f7c-4477-4e80-b0b7-1264f7a28291-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.412664 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412642 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzc7\" (UniqueName: \"kubernetes.io/projected/4dd33f98-33ac-4fa7-b77d-313e19ce5770-kube-api-access-sjzc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.412723 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412704 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd33f98-33ac-4fa7-b77d-313e19ce5770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.412792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412777 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpl9s\" (UniqueName: \"kubernetes.io/projected/e4597027-e561-4878-84e1-82aca3789b86-kube-api-access-xpl9s\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.412827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412817 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c500f-e8bf-485f-b8e2-f3635910d25c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.412861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412845 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hls46\" (UniqueName: \"kubernetes.io/projected/2c181f7c-4477-4e80-b0b7-1264f7a28291-kube-api-access-hls46\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.412904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412877 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvwk\" (UniqueName: \"kubernetes.io/projected/706c500f-e8bf-485f-b8e2-f3635910d25c-kube-api-access-2rvwk\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.412944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.412926 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4597027-e561-4878-84e1-82aca3789b86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.413373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.413342 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c181f7c-4477-4e80-b0b7-1264f7a28291-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.413647 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.413627 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4597027-e561-4878-84e1-82aca3789b86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.413919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.413897 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd33f98-33ac-4fa7-b77d-313e19ce5770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.414153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.414129 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c500f-e8bf-485f-b8e2-f3635910d25c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.487116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.486701 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvwk\" (UniqueName: \"kubernetes.io/projected/706c500f-e8bf-485f-b8e2-f3635910d25c-kube-api-access-2rvwk\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.487116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.486863 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpl9s\" (UniqueName: \"kubernetes.io/projected/e4597027-e561-4878-84e1-82aca3789b86-kube-api-access-xpl9s\") pod \"maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.487387 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.487341 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzc7\" (UniqueName: \"kubernetes.io/projected/4dd33f98-33ac-4fa7-b77d-313e19ce5770-kube-api-access-sjzc7\") pod \"maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.488755 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.488435 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hls46\" (UniqueName: \"kubernetes.io/projected/2c181f7c-4477-4e80-b0b7-1264f7a28291-kube-api-access-hls46\") pod \"maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.584030 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.583904 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:30:00.584520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.584501 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:00.585566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.585537 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:00.585735 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.585645 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:00.588137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.588117 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:00.613716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.612677 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:00.919906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:00.919873 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:01.238210 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:01.238178 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:01.404414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:01.404391 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp"] Nov 03 07:30:01.404858 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:30:01.404826 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4597027_e561_4878_84e1_82aca3789b86.slice/crio-19f180957d5e151d78ed76236a03a0a38bbbf826eda983a6d2d5c159f4e37a23 WatchSource:0}: Error finding container 19f180957d5e151d78ed76236a03a0a38bbbf826eda983a6d2d5c159f4e37a23: Status 404 returned error can't find the container with id 19f180957d5e151d78ed76236a03a0a38bbbf826eda983a6d2d5c159f4e37a23 Nov 03 07:30:01.429479 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:01.429449 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:01.443293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:01.443264 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" event={"ID":"e4597027-e561-4878-84e1-82aca3789b86","Type":"ContainerStarted","Data":"19f180957d5e151d78ed76236a03a0a38bbbf826eda983a6d2d5c159f4e37a23"} Nov 03 07:30:01.469200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:01.469163 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9"] Nov 03 07:30:01.473963 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:30:01.473938 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c181f7c_4477_4e80_b0b7_1264f7a28291.slice/crio-ead56430f91f4133177601aaac86ef05450231142ec5b5cf014479ac7b9561c6 WatchSource:0}: Error finding container ead56430f91f4133177601aaac86ef05450231142ec5b5cf014479ac7b9561c6: Status 404 returned error can't find the container with id ead56430f91f4133177601aaac86ef05450231142ec5b5cf014479ac7b9561c6 Nov 03 07:30:01.517161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:01.517119 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr"] Nov 03 07:30:01.530618 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:30:01.530575 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706c500f_e8bf_485f_b8e2_f3635910d25c.slice/crio-0a0f5f459c28c5c5a0a1a4460ba17155f7d90defec36b30b78a5f04001b6292a WatchSource:0}: Error finding container 0a0f5f459c28c5c5a0a1a4460ba17155f7d90defec36b30b78a5f04001b6292a: Status 404 returned error can't find the container with id 0a0f5f459c28c5c5a0a1a4460ba17155f7d90defec36b30b78a5f04001b6292a Nov 03 07:30:01.534158 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:30:01.534129 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd33f98_33ac_4fa7_b77d_313e19ce5770.slice/crio-a1eb7cae67767db25680701a72c42ed7f758a8470836667bd5ab0177e8b5d9b9 WatchSource:0}: Error finding container a1eb7cae67767db25680701a72c42ed7f758a8470836667bd5ab0177e8b5d9b9: Status 404 returned error can't find the container with id a1eb7cae67767db25680701a72c42ed7f758a8470836667bd5ab0177e8b5d9b9 Nov 03 07:30:01.534658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:01.534633 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n"] Nov 03 07:30:02.459000 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.458964 4925 generic.go:356] "Generic (PLEG): container finished" podID="e4597027-e561-4878-84e1-82aca3789b86" containerID="7a21a6cb0c8861b6fc48866935564fd763f49da7f54a1b6992d9f38e2b1fb7d7" exitCode=60 Nov 03 07:30:02.459506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.459070 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" event={"ID":"e4597027-e561-4878-84e1-82aca3789b86","Type":"ContainerDied","Data":"7a21a6cb0c8861b6fc48866935564fd763f49da7f54a1b6992d9f38e2b1fb7d7"} Nov 03 07:30:02.459567 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.459551 4925 scope.go:122] "RemoveContainer" containerID="7a21a6cb0c8861b6fc48866935564fd763f49da7f54a1b6992d9f38e2b1fb7d7" Nov 03 07:30:02.465613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.465576 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerStarted","Data":"dde69a1afdd9e430527079d260cf3012cd0b584bb52743c7eb6422ed72b0394b"} Nov 03 07:30:02.465742 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.465637 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerStarted","Data":"0a0f5f459c28c5c5a0a1a4460ba17155f7d90defec36b30b78a5f04001b6292a"} Nov 03 07:30:02.471453 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.471426 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" event={"ID":"2c181f7c-4477-4e80-b0b7-1264f7a28291","Type":"ContainerStarted","Data":"689b7d4a41d231863c10cb4e51a5ff7abe560fe0d635a7a059f6d54865df8241"} Nov 03 07:30:02.471581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.471491 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" event={"ID":"2c181f7c-4477-4e80-b0b7-1264f7a28291","Type":"ContainerStarted","Data":"ead56430f91f4133177601aaac86ef05450231142ec5b5cf014479ac7b9561c6"} Nov 03 07:30:02.473644 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.473620 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerStarted","Data":"b0228903b5a43cd7b6b271f5df4eafdb7ba7d932ed9ca87fc1605bc408a5e473"} Nov 03 07:30:02.473767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.473659 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerStarted","Data":"a1eb7cae67767db25680701a72c42ed7f758a8470836667bd5ab0177e8b5d9b9"} Nov 03 07:30:02.701834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.701782 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" podStartSLOduration=2.701768431 podStartE2EDuration="2.701768431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:02.699277723 +0000 UTC m=+1862.632806379" watchObservedRunningTime="2026-11-03 07:30:02.701768431 +0000 UTC m=+1862.635297087" Nov 03 07:30:02.820460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.820406 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" podStartSLOduration=2.820393219 podStartE2EDuration="2.820393219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:02.817852925 +0000 UTC m=+1862.751381582" watchObservedRunningTime="2026-11-03 07:30:02.820393219 +0000 UTC m=+1862.753921922" Nov 03 07:30:02.938710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:02.938658 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" podStartSLOduration=2.938644337 podStartE2EDuration="2.938644337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:02.936805569 +0000 UTC m=+1862.870334226" watchObservedRunningTime="2026-11-03 07:30:02.938644337 +0000 UTC m=+1862.872172992" Nov 03 07:30:03.485360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.485318 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" event={"ID":"e4597027-e561-4878-84e1-82aca3789b86","Type":"ContainerStarted","Data":"a00eacd75f9ca8f9f4bf7780fa8577f82b6835042463624d8ada962e8d63f8f1"} Nov 03 07:30:03.487550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.487524 4925 generic.go:356] "Generic (PLEG): container finished" podID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerID="dde69a1afdd9e430527079d260cf3012cd0b584bb52743c7eb6422ed72b0394b" exitCode=60 Nov 03 07:30:03.487770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.487604 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerDied","Data":"dde69a1afdd9e430527079d260cf3012cd0b584bb52743c7eb6422ed72b0394b"} Nov 03 07:30:03.487885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.487865 4925 scope.go:122] "RemoveContainer" containerID="dde69a1afdd9e430527079d260cf3012cd0b584bb52743c7eb6422ed72b0394b" Nov 03 07:30:03.490082 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.490059 4925 generic.go:356] "Generic (PLEG): container finished" podID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerID="689b7d4a41d231863c10cb4e51a5ff7abe560fe0d635a7a059f6d54865df8241" exitCode=60 Nov 03 07:30:03.490192 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.490141 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" event={"ID":"2c181f7c-4477-4e80-b0b7-1264f7a28291","Type":"ContainerDied","Data":"689b7d4a41d231863c10cb4e51a5ff7abe560fe0d635a7a059f6d54865df8241"} Nov 03 07:30:03.490450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.490429 4925 scope.go:122] "RemoveContainer" containerID="689b7d4a41d231863c10cb4e51a5ff7abe560fe0d635a7a059f6d54865df8241" Nov 03 07:30:03.494908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.494810 4925 generic.go:356] "Generic (PLEG): container finished" podID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerID="b0228903b5a43cd7b6b271f5df4eafdb7ba7d932ed9ca87fc1605bc408a5e473" exitCode=60 Nov 03 07:30:03.494908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.494862 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerDied","Data":"b0228903b5a43cd7b6b271f5df4eafdb7ba7d932ed9ca87fc1605bc408a5e473"} Nov 03 07:30:03.495206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.495189 4925 scope.go:122] "RemoveContainer" containerID="b0228903b5a43cd7b6b271f5df4eafdb7ba7d932ed9ca87fc1605bc408a5e473" Nov 03 07:30:03.608290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:03.608239 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" podStartSLOduration=3.608226385 podStartE2EDuration="3.608226385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:03.605577014 +0000 UTC m=+1863.539105671" watchObservedRunningTime="2026-11-03 07:30:03.608226385 +0000 UTC m=+1863.541755041" Nov 03 07:30:04.504463 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:04.504425 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerStarted","Data":"4ab939144d3b2a091296a63fdfbc0731e748a3fc7d3f4c61f35526f955870cdc"} Nov 03 07:30:04.506660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:04.506632 4925 generic.go:356] "Generic (PLEG): container finished" podID="e4597027-e561-4878-84e1-82aca3789b86" containerID="a00eacd75f9ca8f9f4bf7780fa8577f82b6835042463624d8ada962e8d63f8f1" exitCode=60 Nov 03 07:30:04.506826 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:04.506720 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" event={"ID":"e4597027-e561-4878-84e1-82aca3789b86","Type":"ContainerDied","Data":"a00eacd75f9ca8f9f4bf7780fa8577f82b6835042463624d8ada962e8d63f8f1"} Nov 03 07:30:04.506826 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:04.506783 4925 scope.go:122] "RemoveContainer" containerID="7a21a6cb0c8861b6fc48866935564fd763f49da7f54a1b6992d9f38e2b1fb7d7" Nov 03 07:30:04.506995 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:04.506979 4925 scope.go:122] "RemoveContainer" containerID="a00eacd75f9ca8f9f4bf7780fa8577f82b6835042463624d8ada962e8d63f8f1" Nov 03 07:30:04.507397 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:04.507371 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp_odh-ai-gateway-infra(e4597027-e561-4878-84e1-82aca3789b86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" podUID="e4597027-e561-4878-84e1-82aca3789b86" Nov 03 07:30:04.508924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:04.508898 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerStarted","Data":"f8e2cd9269ad3aeb0ccb409d1beddd86637b4464ade201bd89c156dacf7680cf"} Nov 03 07:30:04.510897 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:04.510871 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" event={"ID":"2c181f7c-4477-4e80-b0b7-1264f7a28291","Type":"ContainerStarted","Data":"cad71cd94d26715713b6898fbe01db7736da9dd292a39344743d3aea430dd7a5"} Nov 03 07:30:05.441032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.440987 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:30:05.520252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.520217 4925 generic.go:356] "Generic (PLEG): container finished" podID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerID="4ab939144d3b2a091296a63fdfbc0731e748a3fc7d3f4c61f35526f955870cdc" exitCode=60 Nov 03 07:30:05.520634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.520297 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerDied","Data":"4ab939144d3b2a091296a63fdfbc0731e748a3fc7d3f4c61f35526f955870cdc"} Nov 03 07:30:05.520634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.520357 4925 scope.go:122] "RemoveContainer" containerID="b0228903b5a43cd7b6b271f5df4eafdb7ba7d932ed9ca87fc1605bc408a5e473" Nov 03 07:30:05.520634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.520614 4925 scope.go:122] "RemoveContainer" containerID="4ab939144d3b2a091296a63fdfbc0731e748a3fc7d3f4c61f35526f955870cdc" Nov 03 07:30:05.521094 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:05.521065 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n_odh-ai-gateway-infra(4dd33f98-33ac-4fa7-b77d-313e19ce5770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" Nov 03 07:30:05.523786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.523767 4925 scope.go:122] "RemoveContainer" containerID="a00eacd75f9ca8f9f4bf7780fa8577f82b6835042463624d8ada962e8d63f8f1" Nov 03 07:30:05.524430 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:05.524150 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp_odh-ai-gateway-infra(e4597027-e561-4878-84e1-82aca3789b86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" podUID="e4597027-e561-4878-84e1-82aca3789b86" Nov 03 07:30:05.525634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.525614 4925 generic.go:356] "Generic (PLEG): container finished" podID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerID="f8e2cd9269ad3aeb0ccb409d1beddd86637b4464ade201bd89c156dacf7680cf" exitCode=60 Nov 03 07:30:05.525732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.525692 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerDied","Data":"f8e2cd9269ad3aeb0ccb409d1beddd86637b4464ade201bd89c156dacf7680cf"} Nov 03 07:30:05.525944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.525926 4925 scope.go:122] "RemoveContainer" containerID="f8e2cd9269ad3aeb0ccb409d1beddd86637b4464ade201bd89c156dacf7680cf" Nov 03 07:30:05.526271 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:05.526250 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr_odh-ai-gateway-infra(706c500f-e8bf-485f-b8e2-f3635910d25c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" Nov 03 07:30:05.527669 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.527650 4925 generic.go:356] "Generic (PLEG): container finished" podID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerID="cad71cd94d26715713b6898fbe01db7736da9dd292a39344743d3aea430dd7a5" exitCode=60 Nov 03 07:30:05.527758 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.527720 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" event={"ID":"2c181f7c-4477-4e80-b0b7-1264f7a28291","Type":"ContainerDied","Data":"cad71cd94d26715713b6898fbe01db7736da9dd292a39344743d3aea430dd7a5"} Nov 03 07:30:05.527918 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.527903 4925 scope.go:122] "RemoveContainer" containerID="cad71cd94d26715713b6898fbe01db7736da9dd292a39344743d3aea430dd7a5" Nov 03 07:30:05.528189 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:05.528172 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9_odh-ai-gateway-infra(2c181f7c-4477-4e80-b0b7-1264f7a28291)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" Nov 03 07:30:05.534237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.534222 4925 scope.go:122] "RemoveContainer" containerID="dde69a1afdd9e430527079d260cf3012cd0b584bb52743c7eb6422ed72b0394b" Nov 03 07:30:05.548133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.548036 4925 scope.go:122] "RemoveContainer" containerID="689b7d4a41d231863c10cb4e51a5ff7abe560fe0d635a7a059f6d54865df8241" Nov 03 07:30:05.825000 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:05.824960 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:06.119108 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:06.102173 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:06.535936 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:06.535857 4925 scope.go:122] "RemoveContainer" containerID="4ab939144d3b2a091296a63fdfbc0731e748a3fc7d3f4c61f35526f955870cdc" Nov 03 07:30:06.536332 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:06.536203 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n_odh-ai-gateway-infra(4dd33f98-33ac-4fa7-b77d-313e19ce5770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" Nov 03 07:30:06.537968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:06.537946 4925 scope.go:122] "RemoveContainer" containerID="f8e2cd9269ad3aeb0ccb409d1beddd86637b4464ade201bd89c156dacf7680cf" Nov 03 07:30:06.538299 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:06.538273 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr_odh-ai-gateway-infra(706c500f-e8bf-485f-b8e2-f3635910d25c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" Nov 03 07:30:06.539420 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:06.539404 4925 scope.go:122] "RemoveContainer" containerID="cad71cd94d26715713b6898fbe01db7736da9dd292a39344743d3aea430dd7a5" Nov 03 07:30:06.539620 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:06.539599 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9_odh-ai-gateway-infra(2c181f7c-4477-4e80-b0b7-1264f7a28291)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" Nov 03 07:30:09.169090 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:09.168730 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:30:09.178038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:09.176530 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:09.392992 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:09.390570 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:11.746175 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:11.746133 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:14.183848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:14.183813 4925 scope.go:122] "RemoveContainer" containerID="a00eacd75f9ca8f9f4bf7780fa8577f82b6835042463624d8ada962e8d63f8f1" Nov 03 07:30:14.626833 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:14.626798 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" event={"ID":"e4597027-e561-4878-84e1-82aca3789b86","Type":"ContainerStarted","Data":"687c6a827820ce15bb17f04d993d43c3ad84b309350e61774b3f017c413e8b27"} Nov 03 07:30:15.636805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:15.636771 4925 generic.go:356] "Generic (PLEG): container finished" podID="e4597027-e561-4878-84e1-82aca3789b86" containerID="687c6a827820ce15bb17f04d993d43c3ad84b309350e61774b3f017c413e8b27" exitCode=60 Nov 03 07:30:15.637458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:15.636911 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" event={"ID":"e4597027-e561-4878-84e1-82aca3789b86","Type":"ContainerDied","Data":"687c6a827820ce15bb17f04d993d43c3ad84b309350e61774b3f017c413e8b27"} Nov 03 07:30:15.637458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:15.636969 4925 scope.go:122] "RemoveContainer" containerID="a00eacd75f9ca8f9f4bf7780fa8577f82b6835042463624d8ada962e8d63f8f1" Nov 03 07:30:15.637592 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:15.637514 4925 scope.go:122] "RemoveContainer" containerID="687c6a827820ce15bb17f04d993d43c3ad84b309350e61774b3f017c413e8b27" Nov 03 07:30:15.637927 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:15.637899 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp_odh-ai-gateway-infra(e4597027-e561-4878-84e1-82aca3789b86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" podUID="e4597027-e561-4878-84e1-82aca3789b86" Nov 03 07:30:15.761840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:15.761806 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp"] Nov 03 07:30:16.183953 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.183920 4925 scope.go:122] "RemoveContainer" containerID="cad71cd94d26715713b6898fbe01db7736da9dd292a39344743d3aea430dd7a5" Nov 03 07:30:16.184163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.184140 4925 scope.go:122] "RemoveContainer" containerID="4ab939144d3b2a091296a63fdfbc0731e748a3fc7d3f4c61f35526f955870cdc" Nov 03 07:30:16.649780 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.649747 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerStarted","Data":"2db338848a92f42014c4818e963c4be52b0b915446c0190e0d46d473fb198bd7"} Nov 03 07:30:16.804616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.804514 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:16.860922 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.860897 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:16.885941 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.885912 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4597027-e561-4878-84e1-82aca3789b86-kube-api-access-xpl9s\" (UniqueName: \"kubernetes.io/projected/e4597027-e561-4878-84e1-82aca3789b86-kube-api-access-xpl9s\") pod \"e4597027-e561-4878-84e1-82aca3789b86\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " Nov 03 07:30:16.886218 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.886195 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4597027-e561-4878-84e1-82aca3789b86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4597027-e561-4878-84e1-82aca3789b86-maas-api-service-ca\") pod \"e4597027-e561-4878-84e1-82aca3789b86\" (UID: \"e4597027-e561-4878-84e1-82aca3789b86\") " Nov 03 07:30:16.886539 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.886517 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4597027-e561-4878-84e1-82aca3789b86-maas-api-service-ca" pod "e4597027-e561-4878-84e1-82aca3789b86" (UID: "e4597027-e561-4878-84e1-82aca3789b86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:30:16.886807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.886712 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4597027-e561-4878-84e1-82aca3789b86-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:16.888246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.888217 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4597027-e561-4878-84e1-82aca3789b86-kube-api-access-xpl9s" pod "e4597027-e561-4878-84e1-82aca3789b86" (UID: "e4597027-e561-4878-84e1-82aca3789b86"). InnerVolumeSpecName "kube-api-access-xpl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:30:16.988848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:16.988770 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpl9s\" (UniqueName: \"kubernetes.io/projected/e4597027-e561-4878-84e1-82aca3789b86-kube-api-access-xpl9s\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:17.658089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.658051 4925 generic.go:356] "Generic (PLEG): container finished" podID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerID="2db338848a92f42014c4818e963c4be52b0b915446c0190e0d46d473fb198bd7" exitCode=60 Nov 03 07:30:17.658532 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.658128 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerDied","Data":"2db338848a92f42014c4818e963c4be52b0b915446c0190e0d46d473fb198bd7"} Nov 03 07:30:17.658532 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.658191 4925 scope.go:122] "RemoveContainer" containerID="4ab939144d3b2a091296a63fdfbc0731e748a3fc7d3f4c61f35526f955870cdc" Nov 03 07:30:17.658653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.658620 4925 scope.go:122] "RemoveContainer" containerID="2db338848a92f42014c4818e963c4be52b0b915446c0190e0d46d473fb198bd7" Nov 03 07:30:17.658991 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:17.658968 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n_odh-ai-gateway-infra(4dd33f98-33ac-4fa7-b77d-313e19ce5770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" Nov 03 07:30:17.659812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.659798 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" Nov 03 07:30:17.659890 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.659823 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp" event={"ID":"e4597027-e561-4878-84e1-82aca3789b86","Type":"ContainerDied","Data":"19f180957d5e151d78ed76236a03a0a38bbbf826eda983a6d2d5c159f4e37a23"} Nov 03 07:30:17.661985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.661953 4925 generic.go:356] "Generic (PLEG): container finished" podID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerID="24a5f30f1e7cff5211c83a00e1f0e0bb152c3a51da25a6e7333400c07d6c4b66" exitCode=60 Nov 03 07:30:17.662076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.661989 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" event={"ID":"2c181f7c-4477-4e80-b0b7-1264f7a28291","Type":"ContainerDied","Data":"24a5f30f1e7cff5211c83a00e1f0e0bb152c3a51da25a6e7333400c07d6c4b66"} Nov 03 07:30:17.662442 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.662424 4925 scope.go:122] "RemoveContainer" containerID="24a5f30f1e7cff5211c83a00e1f0e0bb152c3a51da25a6e7333400c07d6c4b66" Nov 03 07:30:17.662774 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:17.662752 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9_odh-ai-gateway-infra(2c181f7c-4477-4e80-b0b7-1264f7a28291)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" Nov 03 07:30:17.671820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.671803 4925 scope.go:122] "RemoveContainer" containerID="687c6a827820ce15bb17f04d993d43c3ad84b309350e61774b3f017c413e8b27" Nov 03 07:30:17.681759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.681745 4925 scope.go:122] "RemoveContainer" containerID="cad71cd94d26715713b6898fbe01db7736da9dd292a39344743d3aea430dd7a5" Nov 03 07:30:17.900910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.900868 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp"] Nov 03 07:30:17.907476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.907449 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ffee69-29894850-j7dvp"] Nov 03 07:30:17.914628 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:17.914542 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n"] Nov 03 07:30:18.184465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.184370 4925 scope.go:122] "RemoveContainer" containerID="f8e2cd9269ad3aeb0ccb409d1beddd86637b4464ade201bd89c156dacf7680cf" Nov 03 07:30:18.193952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.193923 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4597027-e561-4878-84e1-82aca3789b86" path="/var/lib/kubelet/pods/e4597027-e561-4878-84e1-82aca3789b86/volumes" Nov 03 07:30:18.673460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.673414 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerStarted","Data":"4cc357f94c4083a23f33d7f2685896f749d632bcaecfb00364032368b60e5c73"} Nov 03 07:30:18.836816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.836787 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:18.914968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.914939 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd33f98-33ac-4fa7-b77d-313e19ce5770-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd33f98-33ac-4fa7-b77d-313e19ce5770-maas-api-service-ca\") pod \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " Nov 03 07:30:18.915112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.914986 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd33f98-33ac-4fa7-b77d-313e19ce5770-kube-api-access-sjzc7\" (UniqueName: \"kubernetes.io/projected/4dd33f98-33ac-4fa7-b77d-313e19ce5770-kube-api-access-sjzc7\") pod \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\" (UID: \"4dd33f98-33ac-4fa7-b77d-313e19ce5770\") " Nov 03 07:30:18.915324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.915283 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd33f98-33ac-4fa7-b77d-313e19ce5770-maas-api-service-ca" pod "4dd33f98-33ac-4fa7-b77d-313e19ce5770" (UID: "4dd33f98-33ac-4fa7-b77d-313e19ce5770"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:30:18.915497 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.915483 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd33f98-33ac-4fa7-b77d-313e19ce5770-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:18.917303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:18.917282 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd33f98-33ac-4fa7-b77d-313e19ce5770-kube-api-access-sjzc7" pod "4dd33f98-33ac-4fa7-b77d-313e19ce5770" (UID: "4dd33f98-33ac-4fa7-b77d-313e19ce5770"). InnerVolumeSpecName "kube-api-access-sjzc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:30:19.017200 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.017170 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjzc7\" (UniqueName: \"kubernetes.io/projected/4dd33f98-33ac-4fa7-b77d-313e19ce5770-kube-api-access-sjzc7\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:19.035465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.035429 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9"] Nov 03 07:30:19.231074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.231048 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:19.320329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.320298 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c181f7c-4477-4e80-b0b7-1264f7a28291-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c181f7c-4477-4e80-b0b7-1264f7a28291-maas-api-service-ca\") pod \"2c181f7c-4477-4e80-b0b7-1264f7a28291\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " Nov 03 07:30:19.320502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.320436 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c181f7c-4477-4e80-b0b7-1264f7a28291-kube-api-access-hls46\" (UniqueName: \"kubernetes.io/projected/2c181f7c-4477-4e80-b0b7-1264f7a28291-kube-api-access-hls46\") pod \"2c181f7c-4477-4e80-b0b7-1264f7a28291\" (UID: \"2c181f7c-4477-4e80-b0b7-1264f7a28291\") " Nov 03 07:30:19.320680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.320657 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c181f7c-4477-4e80-b0b7-1264f7a28291-maas-api-service-ca" pod "2c181f7c-4477-4e80-b0b7-1264f7a28291" (UID: "2c181f7c-4477-4e80-b0b7-1264f7a28291"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:30:19.320962 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.320945 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c181f7c-4477-4e80-b0b7-1264f7a28291-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:19.322824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.322796 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c181f7c-4477-4e80-b0b7-1264f7a28291-kube-api-access-hls46" pod "2c181f7c-4477-4e80-b0b7-1264f7a28291" (UID: "2c181f7c-4477-4e80-b0b7-1264f7a28291"). InnerVolumeSpecName "kube-api-access-hls46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:30:19.422462 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.422425 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hls46\" (UniqueName: \"kubernetes.io/projected/2c181f7c-4477-4e80-b0b7-1264f7a28291-kube-api-access-hls46\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:19.686447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.686361 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" Nov 03 07:30:19.686447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.686399 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n" event={"ID":"4dd33f98-33ac-4fa7-b77d-313e19ce5770","Type":"ContainerDied","Data":"a1eb7cae67767db25680701a72c42ed7f758a8470836667bd5ab0177e8b5d9b9"} Nov 03 07:30:19.686943 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.686459 4925 scope.go:122] "RemoveContainer" containerID="2db338848a92f42014c4818e963c4be52b0b915446c0190e0d46d473fb198bd7" Nov 03 07:30:19.688481 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.688457 4925 generic.go:356] "Generic (PLEG): container finished" podID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerID="4cc357f94c4083a23f33d7f2685896f749d632bcaecfb00364032368b60e5c73" exitCode=60 Nov 03 07:30:19.688632 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.688522 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerDied","Data":"4cc357f94c4083a23f33d7f2685896f749d632bcaecfb00364032368b60e5c73"} Nov 03 07:30:19.688933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.688917 4925 scope.go:122] "RemoveContainer" containerID="4cc357f94c4083a23f33d7f2685896f749d632bcaecfb00364032368b60e5c73" Nov 03 07:30:19.689264 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:19.689242 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr_odh-ai-gateway-infra(706c500f-e8bf-485f-b8e2-f3635910d25c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" Nov 03 07:30:19.691318 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.691293 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" Nov 03 07:30:19.691318 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.691304 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9" event={"ID":"2c181f7c-4477-4e80-b0b7-1264f7a28291","Type":"ContainerDied","Data":"ead56430f91f4133177601aaac86ef05450231142ec5b5cf014479ac7b9561c6"} Nov 03 07:30:19.698176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.698152 4925 scope.go:122] "RemoveContainer" containerID="f8e2cd9269ad3aeb0ccb409d1beddd86637b4464ade201bd89c156dacf7680cf" Nov 03 07:30:19.713069 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.713052 4925 scope.go:122] "RemoveContainer" containerID="24a5f30f1e7cff5211c83a00e1f0e0bb152c3a51da25a6e7333400c07d6c4b66" Nov 03 07:30:19.811348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.811321 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr"] Nov 03 07:30:19.865312 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.865275 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n"] Nov 03 07:30:19.872051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:19.871996 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9697f7-29894850-b8s2n"] Nov 03 07:30:20.117524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.117487 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9"] Nov 03 07:30:20.124483 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.124451 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6606d6-29894850-k2rr9"] Nov 03 07:30:20.204816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.204767 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" path="/var/lib/kubelet/pods/2c181f7c-4477-4e80-b0b7-1264f7a28291/volumes" Nov 03 07:30:20.205420 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.205393 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" path="/var/lib/kubelet/pods/4dd33f98-33ac-4fa7-b77d-313e19ce5770/volumes" Nov 03 07:30:20.864519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.864494 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:20.941464 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.941436 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706c500f-e8bf-485f-b8e2-f3635910d25c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c500f-e8bf-485f-b8e2-f3635910d25c-maas-api-service-ca\") pod \"706c500f-e8bf-485f-b8e2-f3635910d25c\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " Nov 03 07:30:20.941716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.941690 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706c500f-e8bf-485f-b8e2-f3635910d25c-kube-api-access-2rvwk\" (UniqueName: \"kubernetes.io/projected/706c500f-e8bf-485f-b8e2-f3635910d25c-kube-api-access-2rvwk\") pod \"706c500f-e8bf-485f-b8e2-f3635910d25c\" (UID: \"706c500f-e8bf-485f-b8e2-f3635910d25c\") " Nov 03 07:30:20.941918 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.941791 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706c500f-e8bf-485f-b8e2-f3635910d25c-maas-api-service-ca" pod "706c500f-e8bf-485f-b8e2-f3635910d25c" (UID: "706c500f-e8bf-485f-b8e2-f3635910d25c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:30:20.942206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.942184 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/706c500f-e8bf-485f-b8e2-f3635910d25c-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:20.944447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:20.944414 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706c500f-e8bf-485f-b8e2-f3635910d25c-kube-api-access-2rvwk" pod "706c500f-e8bf-485f-b8e2-f3635910d25c" (UID: "706c500f-e8bf-485f-b8e2-f3635910d25c"). InnerVolumeSpecName "kube-api-access-2rvwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:30:21.043802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:21.043708 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvwk\" (UniqueName: \"kubernetes.io/projected/706c500f-e8bf-485f-b8e2-f3635910d25c-kube-api-access-2rvwk\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:21.713190 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:21.713160 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" Nov 03 07:30:21.713360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:21.713197 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr" event={"ID":"706c500f-e8bf-485f-b8e2-f3635910d25c","Type":"ContainerDied","Data":"0a0f5f459c28c5c5a0a1a4460ba17155f7d90defec36b30b78a5f04001b6292a"} Nov 03 07:30:21.713360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:21.713260 4925 scope.go:122] "RemoveContainer" containerID="4cc357f94c4083a23f33d7f2685896f749d632bcaecfb00364032368b60e5c73" Nov 03 07:30:21.887866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:21.887837 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr"] Nov 03 07:30:21.896514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:21.896479 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894850-mx5sr"] Nov 03 07:30:21.908387 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:21.908351 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:22.195773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:22.195735 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" path="/var/lib/kubelet/pods/706c500f-e8bf-485f-b8e2-f3635910d25c/volumes" Nov 03 07:30:26.879598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:26.879561 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:31.946312 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:31.946255 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:34.226475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:34.226437 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:30:37.232917 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:37.232871 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:37.751498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:37.751445 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:30:37.751857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:37.751821 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:30:39.140523 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:30:39.140488 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:30:42.498056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:42.496920 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:47.412988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:47.412947 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:52.565816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:52.565768 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:56.969662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:56.967175 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:57.441202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:57.441165 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:58.064780 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.064744 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:58.778088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778057 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks"] Nov 03 07:30:58.778606 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778592 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:30:58.778659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778607 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:30:58.778659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778621 4925 cpu_manager.go:414] "containerMap: removing container" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778625 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778639 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:30:58.778659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778644 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:30:58.778659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778653 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:30:58.778659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778657 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778675 4925 cpu_manager.go:414] "containerMap: removing container" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778679 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778687 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778691 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778698 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778702 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778710 4925 cpu_manager.go:414] "containerMap: removing container" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778714 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778729 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778734 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778821 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778827 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778834 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778839 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778846 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778852 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778859 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778866 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:30:58.778867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.778872 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:30:58.782368 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.782352 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795339 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-token\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795372 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dgz\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-kube-api-access-p8dgz\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795394 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-socket\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795570 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795559 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-podinfo\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795583 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-envoy\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795665 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-certs\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795684 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210fc062-7755-456f-80d0-c2dc5366b0f9-istiod-ca-cert\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795757 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-credential-socket\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.795812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.795787 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-data\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.839062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.838998 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-32b500-openshift-default-dockercfg-b5g2d\"" Nov 03 07:30:58.894465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.894425 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks"] Nov 03 07:30:58.897786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.897754 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-socket\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.897968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.897900 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-podinfo\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.897968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.897932 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-envoy\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898002 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-certs\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898047 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210fc062-7755-456f-80d0-c2dc5366b0f9-istiod-ca-cert\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898085 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-credential-socket\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898112 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-data\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898189 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-token\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898217 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dgz\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-kube-api-access-p8dgz\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898271 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-socket\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898833 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-data\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.898982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.898945 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210fc062-7755-456f-80d0-c2dc5366b0f9-istiod-ca-cert\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.899166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.899119 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-credential-socket\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.899549 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.899522 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-certs\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.901515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.901485 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-podinfo\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.901677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.901529 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-envoy\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.924275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.924088 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:58.968182 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.968146 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-token\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:58.971720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:58.970842 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dgz\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-kube-api-access-p8dgz\") pod \"e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:59.096878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:59.096841 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:30:59.132291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:59.132253 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:30:59.571159 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:30:59.571128 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210fc062_7755_456f_80d0_c2dc5366b0f9.slice/crio-feb168b99e5022afa551c01e1bf85396f9f3f7bb26cdca1d73821072928094d1 WatchSource:0}: Error finding container feb168b99e5022afa551c01e1bf85396f9f3f7bb26cdca1d73821072928094d1: Status 404 returned error can't find the container with id feb168b99e5022afa551c01e1bf85396f9f3f7bb26cdca1d73821072928094d1 Nov 03 07:30:59.571297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:59.571190 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks"] Nov 03 07:30:59.574465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:59.574426 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:30:59.574586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:59.574515 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:30:59.574586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:30:59.574556 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:31:00.062224 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:00.062130 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" event={"ID":"210fc062-7755-456f-80d0-c2dc5366b0f9","Type":"ContainerStarted","Data":"2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb"} Nov 03 07:31:00.062224 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:00.062188 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" event={"ID":"210fc062-7755-456f-80d0-c2dc5366b0f9","Type":"ContainerStarted","Data":"feb168b99e5022afa551c01e1bf85396f9f3f7bb26cdca1d73821072928094d1"} Nov 03 07:31:00.097351 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:00.097308 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:31:00.103430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:00.103405 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:31:00.239729 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:00.239677 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podStartSLOduration=2.239663255 podStartE2EDuration="2.239663255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:00.236628056 +0000 UTC m=+1920.170156725" watchObservedRunningTime="2026-11-03 07:31:00.239663255 +0000 UTC m=+1920.173191910" Nov 03 07:31:01.068795 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:01.068763 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:31:01.069769 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:01.069747 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:31:01.576136 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:01.576103 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:03.729900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:03.729851 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:03.935392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:03.935355 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:08.857254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:08.857219 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:09.103652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.103611 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:09.147275 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:31:09.147185 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:31:09.304354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.304313 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:09.538861 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.538772 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:09.949302 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.949204 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr"] Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.949935 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.949954 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.949982 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.949987 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.950026 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.950034 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c181f7c-4477-4e80-b0b7-1264f7a28291" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.950171 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd33f98-33ac-4fa7-b77d-313e19ce5770" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.950181 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="706c500f-e8bf-485f-b8e2-f3635910d25c" containerName="cleanup" Nov 03 07:31:09.950239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.950197 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4597027-e561-4878-84e1-82aca3789b86" containerName="cleanup" Nov 03 07:31:09.954930 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:09.954909 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.080044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.072323 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-1fa30f-openshift-default-dockercfg-vlt9f\"" Nov 03 07:31:10.101113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.101004 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:10.101376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.101355 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr"] Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.111951 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-envoy\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112036 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e200ac-14ee-4353-8101-31c908fa3dcd-istiod-ca-cert\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112107 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-data\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112155 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-socket\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112188 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-podinfo\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112224 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-certs\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112309 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whth6\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-kube-api-access-whth6\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112355 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-token\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.112640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.112411 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-credential-socket\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.213960 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.213883 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whth6\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-kube-api-access-whth6\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214186 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214170 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-token\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214277 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-credential-socket\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214417 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-envoy\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214490 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e200ac-14ee-4353-8101-31c908fa3dcd-istiod-ca-cert\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214577 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-data\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214665 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214656 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-socket\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214722 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-podinfo\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.214796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.214787 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-certs\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.215206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.215189 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-certs\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.221352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.221311 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-credential-socket\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.221352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.221331 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-data\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.221779 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.221756 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-socket\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.222083 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.222034 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e200ac-14ee-4353-8101-31c908fa3dcd-istiod-ca-cert\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.224877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.224855 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-envoy\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.226098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.225996 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-podinfo\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.283923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.283813 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-token\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.286999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.286942 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whth6\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-kube-api-access-whth6\") pod \"e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:10.521778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.521669 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:10.585112 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:10.585076 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:11.035473 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:31:11.035443 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e200ac_14ee_4353_8101_31c908fa3dcd.slice/crio-62dcbc04004af03dfb200442de4835b0337a959b2d738fc278adcfac55164d06 WatchSource:0}: Error finding container 62dcbc04004af03dfb200442de4835b0337a959b2d738fc278adcfac55164d06: Status 404 returned error can't find the container with id 62dcbc04004af03dfb200442de4835b0337a959b2d738fc278adcfac55164d06 Nov 03 07:31:11.037041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.037002 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr"] Nov 03 07:31:11.038156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.038121 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:31:11.038245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.038201 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:31:11.038317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.038246 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:31:11.213989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.213953 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" event={"ID":"a1e200ac-14ee-4353-8101-31c908fa3dcd","Type":"ContainerStarted","Data":"72288a1bf896f3839b44379bc0c51b29f54a418ab446c210f96d96367159b759"} Nov 03 07:31:11.214185 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.214041 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" event={"ID":"a1e200ac-14ee-4353-8101-31c908fa3dcd","Type":"ContainerStarted","Data":"62dcbc04004af03dfb200442de4835b0337a959b2d738fc278adcfac55164d06"} Nov 03 07:31:11.341270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.341215 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" podStartSLOduration=2.341201326 podStartE2EDuration="2.341201326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:11.339820904 +0000 UTC m=+1931.273349562" watchObservedRunningTime="2026-11-03 07:31:11.341201326 +0000 UTC m=+1931.274729976" Nov 03 07:31:11.587157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.586545 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:11.587157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:11.586671 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:12.591762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:12.591727 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:12.998476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:12.998386 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:13.228363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:13.228331 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:13.229320 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:13.229296 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:31:14.561986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:14.561948 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:14.898134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:14.898102 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:15.134963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:15.134918 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:15.345045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:15.344195 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:17.956126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:17.956083 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:18.434425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:18.434388 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:18.979535 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:18.979500 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8"] Nov 03 07:31:18.987417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:18.987390 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.043328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.043281 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8e6dc7-openshift-default-dockercfg-j82c2\"" Nov 03 07:31:19.079276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.079237 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj9b7\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-kube-api-access-xj9b7\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.079583 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.079562 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-credential-socket\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.079761 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.079743 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1569787-fd4c-4255-aacf-5d74974813ac-istiod-ca-cert\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.079931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.079914 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1569787-fd4c-4255-aacf-5d74974813ac-istio-podinfo\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.080204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.080184 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-envoy\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.080404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.080390 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-istio-token\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.080631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.080568 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-data\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.086254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.086230 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-certs\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.086428 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.086412 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-socket\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.193968 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1569787-fd4c-4255-aacf-5d74974813ac-istiod-ca-cert\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194064 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1569787-fd4c-4255-aacf-5d74974813ac-istio-podinfo\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194189 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-envoy\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194253 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-istio-token\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194331 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-data\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194370 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-certs\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194407 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-socket\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194534 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj9b7\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-kube-api-access-xj9b7\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.194567 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-credential-socket\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.195106 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-credential-socket\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.195953 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.195842 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1569787-fd4c-4255-aacf-5d74974813ac-istiod-ca-cert\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.228099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.209635 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1569787-fd4c-4255-aacf-5d74974813ac-istio-podinfo\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.229645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.229573 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-certs\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.254854 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.243978 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-data\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.255707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.252130 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-socket\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.256431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.255129 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8"] Nov 03 07:31:19.257772 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.257739 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:19.257923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.256348 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-envoy\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.315946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.315836 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-istio-token\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.330420 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.330348 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj9b7\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-kube-api-access-xj9b7\") pod \"e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.354708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.354412 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:19.585299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.585206 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:19.792113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.791107 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:19.852459 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:31:19.852426 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1569787_fd4c_4255_aacf_5d74974813ac.slice/crio-fe10d8f2820743d3865de04d471581ae6d9c2a3bcae96bbf37f08b130ccebe64 WatchSource:0}: Error finding container fe10d8f2820743d3865de04d471581ae6d9c2a3bcae96bbf37f08b130ccebe64: Status 404 returned error can't find the container with id fe10d8f2820743d3865de04d471581ae6d9c2a3bcae96bbf37f08b130ccebe64 Nov 03 07:31:19.852624 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.852528 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8"] Nov 03 07:31:19.856380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.856315 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:31:19.856496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.856431 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:31:19.856496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.856483 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:31:19.999864 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:19.999812 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:20.137748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:20.137712 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:31:20.252034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:20.248087 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:20.423764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:20.423671 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" event={"ID":"f1569787-fd4c-4255-aacf-5d74974813ac","Type":"ContainerStarted","Data":"a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a"} Nov 03 07:31:20.424052 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:20.423999 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" event={"ID":"f1569787-fd4c-4255-aacf-5d74974813ac","Type":"ContainerStarted","Data":"fe10d8f2820743d3865de04d471581ae6d9c2a3bcae96bbf37f08b130ccebe64"} Nov 03 07:31:20.445335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:20.445301 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:20.599045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:20.596160 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podStartSLOduration=2.596139155 podStartE2EDuration="2.596139155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:20.592989037 +0000 UTC m=+1940.526517694" watchObservedRunningTime="2026-11-03 07:31:20.596139155 +0000 UTC m=+1940.529667814" Nov 03 07:31:21.356977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:21.356945 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:21.363578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:21.363550 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:21.430955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:21.430926 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:21.431912 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:21.431892 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:31:21.509906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:21.509868 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:21.755674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:21.755576 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:24.467475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:24.467437 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:24.679917 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:24.679885 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:26.385140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:26.385108 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:26.585940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:26.585908 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:26.794199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:26.794108 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:29.224161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:29.224125 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:31.541265 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:31.541224 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:31.763032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:31.754047 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:31.950446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:31.950383 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:36.865616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:36.865573 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:37.074312 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:37.074270 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:39.138622 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:31:39.138585 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:31:42.023907 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:42.023860 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:46.844050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:46.842154 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:31:47.096109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:47.095256 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:31:47.279037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:47.278641 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:31:47.362281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:47.362154 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:47.436047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:47.434331 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:31:52.373758 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:52.373718 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:31:57.348326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:31:57.348284 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:01.933250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:01.933210 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:02.432848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.432805 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:02.536938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.536905 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-d3c0bd"] Nov 03 07:32:02.541219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.541201 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:02.596074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.596047 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-d3c0bd"] Nov 03 07:32:02.631070 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.631037 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kn8\" (UniqueName: \"kubernetes.io/projected/db45438b-207c-4cd3-905e-0ee02e4a64d0-kube-api-access-p6kn8\") pod \"test-curl-7135-d3c0bd\" (UID: \"db45438b-207c-4cd3-905e-0ee02e4a64d0\") " pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:02.641798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.641777 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mnr67\"" Nov 03 07:32:02.732620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.732536 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kn8\" (UniqueName: \"kubernetes.io/projected/db45438b-207c-4cd3-905e-0ee02e4a64d0-kube-api-access-p6kn8\") pod \"test-curl-7135-d3c0bd\" (UID: \"db45438b-207c-4cd3-905e-0ee02e4a64d0\") " pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:02.796317 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.796289 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kn8\" (UniqueName: \"kubernetes.io/projected/db45438b-207c-4cd3-905e-0ee02e4a64d0-kube-api-access-p6kn8\") pod \"test-curl-7135-d3c0bd\" (UID: \"db45438b-207c-4cd3-905e-0ee02e4a64d0\") " pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:02.853801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:02.853765 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:03.234895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:03.234860 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-d3c0bd"] Nov 03 07:32:03.235709 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:32:03.235683 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb45438b_207c_4cd3_905e_0ee02e4a64d0.slice/crio-7cd01d627b39a16342e3ae970143fd15fd766dd83578242b9c8de7a83bca1d29 WatchSource:0}: Error finding container 7cd01d627b39a16342e3ae970143fd15fd766dd83578242b9c8de7a83bca1d29: Status 404 returned error can't find the container with id 7cd01d627b39a16342e3ae970143fd15fd766dd83578242b9c8de7a83bca1d29 Nov 03 07:32:03.237784 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:03.237764 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:32:03.469784 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:03.469750 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:32:03.928851 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:03.928815 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-d3c0bd" event={"ID":"db45438b-207c-4cd3-905e-0ee02e4a64d0","Type":"ContainerStarted","Data":"7cd01d627b39a16342e3ae970143fd15fd766dd83578242b9c8de7a83bca1d29"} Nov 03 07:32:06.953554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:06.953510 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-d3c0bd" event={"ID":"db45438b-207c-4cd3-905e-0ee02e4a64d0","Type":"ContainerStarted","Data":"2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253"} Nov 03 07:32:07.073441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:07.073382 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-d3c0bd" podStartSLOduration=2.305610181 podStartE2EDuration="5.073367048s" totalImagesPullingTime="2.767756867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:32:07.071740788 +0000 UTC m=+1987.005269445" watchObservedRunningTime="2026-11-03 07:32:07.073367048 +0000 UTC m=+1987.006895733" Nov 03 07:32:07.502135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:07.502095 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:08.614093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:08.614059 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-d3c0bd"] Nov 03 07:32:08.621910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:08.621876 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-d3c0bd"] Nov 03 07:32:08.967224 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:08.967130 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-d3c0bd" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" containerName="test-curl-7135-d3c0bd" containerID="cri-o://2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253" gracePeriod=2 Nov 03 07:32:09.019695 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:09.019655 4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-d3c0bd\" is forbidden: User \"system:node:ip-10-0-136-184.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-184.us-west-1.compute.internal' and this object" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:09.139954 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:09.139920 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:32:10.239586 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:10.239547 4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-d3c0bd\" is forbidden: User \"system:node:ip-10-0-136-184.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-184.us-west-1.compute.internal' and this object" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:11.223323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:11.223267 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:11.357919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:11.357881 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db45438b-207c-4cd3-905e-0ee02e4a64d0-kube-api-access-p6kn8\" (UniqueName: \"kubernetes.io/projected/db45438b-207c-4cd3-905e-0ee02e4a64d0-kube-api-access-p6kn8\") pod \"db45438b-207c-4cd3-905e-0ee02e4a64d0\" (UID: \"db45438b-207c-4cd3-905e-0ee02e4a64d0\") " Nov 03 07:32:11.360205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:11.360180 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db45438b-207c-4cd3-905e-0ee02e4a64d0-kube-api-access-p6kn8" pod "db45438b-207c-4cd3-905e-0ee02e4a64d0" (UID: "db45438b-207c-4cd3-905e-0ee02e4a64d0"). InnerVolumeSpecName "kube-api-access-p6kn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:32:11.459800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:11.459759 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6kn8\" (UniqueName: \"kubernetes.io/projected/db45438b-207c-4cd3-905e-0ee02e4a64d0-kube-api-access-p6kn8\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:32:11.991191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:11.991153 4925 generic.go:356] "Generic (PLEG): container finished" podID="db45438b-207c-4cd3-905e-0ee02e4a64d0" containerID="2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253" exitCode=137 Nov 03 07:32:11.991397 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:11.991199 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:11.991397 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:11.991214 4925 scope.go:122] "RemoveContainer" containerID="2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253" Nov 03 07:32:12.001807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:12.001788 4925 scope.go:122] "RemoveContainer" containerID="2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253" Nov 03 07:32:12.002143 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:12.002120 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253\": container with ID starting with 2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253 not found: ID does not exist" containerID="2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253" Nov 03 07:32:12.002236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:12.002154 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253"} err="failed to get container status \"2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253\": rpc error: code = NotFound desc = could not find container \"2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253\": container with ID starting with 2dd57225839a602232b86aac81ba55966f65a0f962ce3ed82e21d824283d3253 not found: ID does not exist" Nov 03 07:32:12.052113 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:12.052072 4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-d3c0bd\" is forbidden: User \"system:node:ip-10-0-136-184.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-184.us-west-1.compute.internal' and this object" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" pod="opendatahub/test-curl-7135-d3c0bd" Nov 03 07:32:12.191675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:12.191644 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" path="/var/lib/kubelet/pods/db45438b-207c-4cd3-905e-0ee02e4a64d0/volumes" Nov 03 07:32:12.845428 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:12.845390 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:18.280500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:18.280452 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:22.408344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:22.408308 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:22.614618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:22.614581 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:23.360159 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.360124 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb"] Nov 03 07:32:23.360719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.360702 4925 cpu_manager.go:414] "containerMap: removing container" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" containerName="test-curl-7135-d3c0bd" Nov 03 07:32:23.360719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.360719 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" containerName="test-curl-7135-d3c0bd" Nov 03 07:32:23.360876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.360844 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="db45438b-207c-4cd3-905e-0ee02e4a64d0" containerName="test-curl-7135-d3c0bd" Nov 03 07:32:23.364159 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.364139 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.446272 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.446211 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288b91f6-ab6a-4cba-a40c-b34bc396bf49-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.446806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.446782 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrq6m\" (UniqueName: \"kubernetes.io/projected/288b91f6-ab6a-4cba-a40c-b34bc396bf49-kube-api-access-jrq6m\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.447105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.447077 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.447223 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.447189 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.519035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.476263 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-32b500-serving-cert\"" Nov 03 07:32:23.547979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.547946 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb"] Nov 03 07:32:23.553709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.551294 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.556287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.556247 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.557044 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:23.552524 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-serving-cert: secret "maas-api-e2e-shared-b-w6-32b500-serving-cert" not found Nov 03 07:32:23.557270 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:23.557254 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls podName:288b91f6-ab6a-4cba-a40c-b34bc396bf49 nodeName:}" failed. No retries permitted until 2026-11-03 07:32:24.057229148 +0000 UTC m=+2003.990757801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls") pod "maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" (UID: "288b91f6-ab6a-4cba-a40c-b34bc396bf49") : secret "maas-api-e2e-shared-b-w6-32b500-serving-cert" not found Nov 03 07:32:23.559119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.558554 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288b91f6-ab6a-4cba-a40c-b34bc396bf49-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.568511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.568488 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrq6m\" (UniqueName: \"kubernetes.io/projected/288b91f6-ab6a-4cba-a40c-b34bc396bf49-kube-api-access-jrq6m\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.569713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.567276 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288b91f6-ab6a-4cba-a40c-b34bc396bf49-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.593968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.593937 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.624480 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.624385 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:23.640334 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.640178 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrq6m\" (UniqueName: \"kubernetes.io/projected/288b91f6-ab6a-4cba-a40c-b34bc396bf49-kube-api-access-jrq6m\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:23.823090 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:23.823005 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:24.067993 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:24.067954 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:24.090518 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:24.090479 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:24.093264 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:24.093238 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:24.278539 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:24.278508 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:24.668174 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:24.668145 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb"] Nov 03 07:32:24.668796 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:32:24.668772 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288b91f6_ab6a_4cba_a40c_b34bc396bf49.slice/crio-c32e8ec1719438cbb67f22f4209c8fd8ac4bf88a18fa46f3ad7324124198e7fb WatchSource:0}: Error finding container c32e8ec1719438cbb67f22f4209c8fd8ac4bf88a18fa46f3ad7324124198e7fb: Status 404 returned error can't find the container with id c32e8ec1719438cbb67f22f4209c8fd8ac4bf88a18fa46f3ad7324124198e7fb Nov 03 07:32:25.108306 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:25.108267 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" event={"ID":"288b91f6-ab6a-4cba-a40c-b34bc396bf49","Type":"ContainerStarted","Data":"c32e8ec1719438cbb67f22f4209c8fd8ac4bf88a18fa46f3ad7324124198e7fb"} Nov 03 07:32:26.117441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:26.117377 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" event={"ID":"288b91f6-ab6a-4cba-a40c-b34bc396bf49","Type":"ContainerStarted","Data":"fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d"} Nov 03 07:32:26.117947 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:26.117600 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:26.244534 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:26.244477 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" podStartSLOduration=2.371696279 podStartE2EDuration="3.244464048s" totalImagesPullingTime="872.767769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:32:26.24079392 +0000 UTC m=+2006.174322576" watchObservedRunningTime="2026-11-03 07:32:26.244464048 +0000 UTC m=+2006.177992703" Nov 03 07:32:29.339033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:29.338977 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:32.132369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:32.132340 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:32:34.566300 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:34.566172 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:39.140435 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:39.140399 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:32:39.489971 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:39.489796 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:39.720280 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:39.720242 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:44.624087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:44.620995 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:46.462238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:46.462206 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:32:49.483755 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.483721 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4"] Nov 03 07:32:49.491274 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.491253 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.579585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.579549 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4"] Nov 03 07:32:49.594150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.594113 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-72a642-serving-cert\"" Nov 03 07:32:49.599906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.599872 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.600084 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.599994 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-metrics-certs\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.600084 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.600081 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crkg\" (UniqueName: \"kubernetes.io/projected/7fc9b45c-1517-47f2-959d-9c7c59069398-kube-api-access-9crkg\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.600218 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.600160 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc9b45c-1517-47f2-959d-9c7c59069398-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.702092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.702052 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.702289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.702148 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-metrics-certs\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.702289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.702201 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crkg\" (UniqueName: \"kubernetes.io/projected/7fc9b45c-1517-47f2-959d-9c7c59069398-kube-api-access-9crkg\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.702289 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:49.702218 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-serving-cert: secret "maas-api-e2e-inf-a-72a642-serving-cert" not found Nov 03 07:32:49.702289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.702231 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc9b45c-1517-47f2-959d-9c7c59069398-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.702513 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:32:49.702299 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls podName:7fc9b45c-1517-47f2-959d-9c7c59069398 nodeName:}" failed. No retries permitted until 2026-11-03 07:32:50.202276415 +0000 UTC m=+2030.135805067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls") pod "maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" (UID: "7fc9b45c-1517-47f2-959d-9c7c59069398") : secret "maas-api-e2e-inf-a-72a642-serving-cert" not found Nov 03 07:32:49.702910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.702892 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc9b45c-1517-47f2-959d-9c7c59069398-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.704805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.704784 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-metrics-certs\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.730313 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.730275 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:49.793919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.793826 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crkg\" (UniqueName: \"kubernetes.io/projected/7fc9b45c-1517-47f2-959d-9c7c59069398-kube-api-access-9crkg\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:49.924594 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:49.924557 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:50.113439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:50.113395 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:50.209240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:50.209208 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:50.211744 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:50.211716 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls\") pod \"maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:50.406289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:50.406190 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:50.839490 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:32:50.839458 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc9b45c_1517_47f2_959d_9c7c59069398.slice/crio-293fea73cf687a85b218ce1b5de0d2629ef40464b542ca05019bb1dabe2f178d WatchSource:0}: Error finding container 293fea73cf687a85b218ce1b5de0d2629ef40464b542ca05019bb1dabe2f178d: Status 404 returned error can't find the container with id 293fea73cf687a85b218ce1b5de0d2629ef40464b542ca05019bb1dabe2f178d Nov 03 07:32:50.840979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:50.840955 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4"] Nov 03 07:32:51.309875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:51.309792 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" event={"ID":"7fc9b45c-1517-47f2-959d-9c7c59069398","Type":"ContainerStarted","Data":"293fea73cf687a85b218ce1b5de0d2629ef40464b542ca05019bb1dabe2f178d"} Nov 03 07:32:52.320244 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:52.320206 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" event={"ID":"7fc9b45c-1517-47f2-959d-9c7c59069398","Type":"ContainerStarted","Data":"fee7f056cde181231666e3320d4b605475b29ef3a75cfeb42be2ae759c4e532a"} Nov 03 07:32:52.320762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:52.320274 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:52.445260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:52.445204 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" podStartSLOduration=2.650078585 podStartE2EDuration="3.445184272s" totalImagesPullingTime="795.105687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:32:52.44224575 +0000 UTC m=+2032.375774408" watchObservedRunningTime="2026-11-03 07:32:52.445184272 +0000 UTC m=+2032.378712929" Nov 03 07:32:55.037700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:55.037661 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:32:58.334566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:58.334531 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:32:58.369204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:58.369175 4925 generic.go:356] "Generic (PLEG): container finished" podID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerID="2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb" exitCode=0 Nov 03 07:32:58.369365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:58.369248 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" event={"ID":"210fc062-7755-456f-80d0-c2dc5366b0f9","Type":"ContainerDied","Data":"2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb"} Nov 03 07:32:58.369608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:58.369594 4925 scope.go:122] "RemoveContainer" containerID="2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb" Nov 03 07:32:58.371159 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:58.371131 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:32:58.371260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:58.371196 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:32:58.371260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:58.371228 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:32:59.427036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:32:59.426746 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" event={"ID":"210fc062-7755-456f-80d0-c2dc5366b0f9","Type":"ContainerStarted","Data":"3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab"} Nov 03 07:33:00.231085 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:00.209190 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:33:00.231085 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:00.209266 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 07:33:00.231085 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:00.225788 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:00.241618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:00.241585 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:33:00.473036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:00.468047 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:00.677659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:00.677617 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:01.072203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:01.072162 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:33:01.105045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:01.102437 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:33:01.105045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:01.102509 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 07:33:01.993921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:01.993834 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:02.107048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:02.105436 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:33:02.107048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:02.105511 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 07:33:02.192126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:02.192090 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:02.420375 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:02.420337 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:02.625562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:02.625531 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:03.097598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:03.097553 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:33:03.098140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:03.097637 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 07:33:04.097692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:04.097648 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:33:04.098241 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:04.097717 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 07:33:05.102641 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:05.102612 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:33:05.103922 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:05.103898 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:33:06.394570 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:06.394533 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:06.821388 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:06.821351 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:07.032942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:07.032869 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:07.231107 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:07.231003 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:09.139178 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:33:09.139146 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:33:11.878335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:11.878293 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:12.968505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:12.968459 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:13.159916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:13.159200 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:17.108154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:17.108114 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:17.315081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:17.315040 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:17.555560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:17.554784 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:23.172780 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:23.172741 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:25.744425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:25.744340 4925 generic.go:356] "Generic (PLEG): container finished" podID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerID="72288a1bf896f3839b44379bc0c51b29f54a418ab446c210f96d96367159b759" exitCode=0 Nov 03 07:33:25.744828 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:25.744423 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" event={"ID":"a1e200ac-14ee-4353-8101-31c908fa3dcd","Type":"ContainerDied","Data":"72288a1bf896f3839b44379bc0c51b29f54a418ab446c210f96d96367159b759"} Nov 03 07:33:25.744871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:25.744830 4925 scope.go:122] "RemoveContainer" containerID="72288a1bf896f3839b44379bc0c51b29f54a418ab446c210f96d96367159b759" Nov 03 07:33:25.746603 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:25.746575 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:33:25.746681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:25.746648 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:33:25.746717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:25.746681 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:33:26.783493 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:26.783457 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" event={"ID":"a1e200ac-14ee-4353-8101-31c908fa3dcd","Type":"ContainerStarted","Data":"4eb36bb8fc296e9b6925b3735ec38db1ff86f003dcec40625b14121e4747fa18"} Nov 03 07:33:27.585299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:27.585263 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:33:27.586939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:27.586909 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 07:33:27.587127 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:27.586960 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 07:33:28.228574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:28.228544 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:33:28.585740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:28.585699 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 07:33:28.585948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:28.585779 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 07:33:28.900931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:28.900834 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb"] Nov 03 07:33:28.901843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:28.901800 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" containerID="cri-o://4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905" gracePeriod=30 Nov 03 07:33:28.912955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:28.912922 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:29.014099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:29.014061 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr"] Nov 03 07:33:29.014369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:29.014329 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" containerID="cri-o://bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818" gracePeriod=30 Nov 03 07:33:29.020611 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:29.020584 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg"] Nov 03 07:33:29.021128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:29.021095 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" containerID="cri-o://dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c" gracePeriod=30 Nov 03 07:33:29.585742 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:29.585711 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 07:33:29.586225 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:29.585763 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 07:33:30.301648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:30.301604 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:33:30.301831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:30.301674 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:33:30.363247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:30.363205 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:33:30.363554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:30.363523 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:33:30.585432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:30.585395 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 07:33:30.585619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:30.585459 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 07:33:31.333056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:31.332990 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:31.377923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:31.377880 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:33:31.378152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:31.377947 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:33:31.533615 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:31.533578 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:31.586037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:31.585935 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 07:33:31.586037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:31.585991 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 07:33:32.591323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:32.591282 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:33:32.592476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:32.592454 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:33:34.190592 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.190327 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:33:34.330520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.330474 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-envoy\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.330731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.330554 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-token\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.330731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.330636 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-podinfo\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.330866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.330730 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-socket\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.330866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.330777 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-kube-api-access-5qch5\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-kube-api-access-5qch5\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.330866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.330816 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-data\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.330866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.330862 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istiod-ca-cert\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.331111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.331001 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-credential-socket\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.331111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.331092 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-certs\") pod \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\" (UID: \"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd\") " Nov 03 07:33:34.331988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.331559 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-socket" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.331988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.331887 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-data" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.334352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.333610 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istiod-ca-cert" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:33:34.334880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.334857 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-credential-socket" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.334997 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.334978 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-kube-api-access-5qch5" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "kube-api-access-5qch5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:34.335310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.335283 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-certs" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.350494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.338802 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:33:34.350494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.344930 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.350494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.344962 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.350494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.346470 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-podinfo" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:33:34.353073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.353003 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-envoy" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.357854 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.357827 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-token" pod "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" (UID: "7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:34.365482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.365454 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:33:34.433097 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.433056 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:34.446356 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446274 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004ae909-4d9d-4755-be60-cc0b651ebdcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004ae909-4d9d-4755-be60-cc0b651ebdcd-istiod-ca-cert\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446356 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446322 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-socket\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446544 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446371 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-kube-api-access-smxg9\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-kube-api-access-smxg9\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446544 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446418 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-data\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446544 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446452 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-credential-socket\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446697 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446672 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004ae909-4d9d-4755-be60-cc0b651ebdcd-istiod-ca-cert" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:33:34.446772 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446697 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-podinfo\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446772 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446739 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-certs\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446798 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-token\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-token\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.446946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446881 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-envoy\") pod \"004ae909-4d9d-4755-be60-cc0b651ebdcd\" (UID: \"004ae909-4d9d-4755-be60-cc0b651ebdcd\") " Nov 03 07:33:34.447043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446964 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-credential-socket" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.447043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446995 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-socket" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.447460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447172 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-certs" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.447460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.446996 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-data" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.447624 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447591 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qch5\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-kube-api-access-5qch5\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447624 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447614 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447630 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447645 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447659 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447673 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447724 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004ae909-4d9d-4755-be60-cc0b651ebdcd-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447741 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.447748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447755 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.448303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447788 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.448303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447803 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.448303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.447817 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.449666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.449639 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-envoy" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.450135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.450116 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-podinfo" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:33:34.450234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.450210 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-token" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:34.450852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.450835 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-kube-api-access-smxg9" pod "004ae909-4d9d-4755-be60-cc0b651ebdcd" (UID: "004ae909-4d9d-4755-be60-cc0b651ebdcd"). InnerVolumeSpecName "kube-api-access-smxg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:34.549063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.548802 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-kube-api-access-m2x9n\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-kube-api-access-m2x9n\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.548872 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-envoy\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.548986 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-certs\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.549061 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d32cd9-4b79-46cc-b807-0e183522f514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d32cd9-4b79-46cc-b807-0e183522f514-istiod-ca-cert\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.549105 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-data\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.549166 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-socket\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.549214 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89d32cd9-4b79-46cc-b807-0e183522f514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d32cd9-4b79-46cc-b807-0e183522f514-istio-podinfo\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.549253 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-credential-socket\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.549516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.549276 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-istio-token\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-istio-token\") pod \"89d32cd9-4b79-46cc-b807-0e183522f514\" (UID: \"89d32cd9-4b79-46cc-b807-0e183522f514\") " Nov 03 07:33:34.550656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.550176 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxg9\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-kube-api-access-smxg9\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.550656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.550207 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.550656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.550223 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.550656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.550239 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004ae909-4d9d-4755-be60-cc0b651ebdcd-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.550656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.550382 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-credential-socket" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.551428 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.550914 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-certs" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.551428 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.551394 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d32cd9-4b79-46cc-b807-0e183522f514-istiod-ca-cert" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:33:34.551641 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.551595 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-data" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.551724 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.551665 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-socket" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.551837 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.551818 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-kube-api-access-m2x9n" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "kube-api-access-m2x9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:34.552858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.552812 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-envoy" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:33:34.553730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.553708 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89d32cd9-4b79-46cc-b807-0e183522f514-istio-podinfo" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:33:34.554451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.554430 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-istio-token" pod "89d32cd9-4b79-46cc-b807-0e183522f514" (UID: "89d32cd9-4b79-46cc-b807-0e183522f514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:34.651863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651821 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d32cd9-4b79-46cc-b807-0e183522f514-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.651863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651866 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.652054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651882 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.652054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651896 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2x9n\" (UniqueName: \"kubernetes.io/projected/89d32cd9-4b79-46cc-b807-0e183522f514-kube-api-access-m2x9n\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.652054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651912 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.652054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651925 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.652054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651938 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d32cd9-4b79-46cc-b807-0e183522f514-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.652054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651953 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.652054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.651968 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d32cd9-4b79-46cc-b807-0e183522f514-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:34.843689 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.843657 4925 generic.go:356] "Generic (PLEG): container finished" podID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerID="4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905" exitCode=0 Nov 03 07:33:34.843832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.843714 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" Nov 03 07:33:34.843832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.843740 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" event={"ID":"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd","Type":"ContainerDied","Data":"4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905"} Nov 03 07:33:34.843832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.843810 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb" event={"ID":"7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd","Type":"ContainerDied","Data":"3b22ba13105f074bb200bb9f86165af2cc4b6815e34ad228545fa01bca43af2c"} Nov 03 07:33:34.843832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.843831 4925 scope.go:122] "RemoveContainer" containerID="4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905" Nov 03 07:33:34.845412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.845392 4925 generic.go:356] "Generic (PLEG): container finished" podID="89d32cd9-4b79-46cc-b807-0e183522f514" containerID="dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c" exitCode=0 Nov 03 07:33:34.845536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.845433 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" Nov 03 07:33:34.845591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.845506 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" event={"ID":"89d32cd9-4b79-46cc-b807-0e183522f514","Type":"ContainerDied","Data":"dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c"} Nov 03 07:33:34.845591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.845576 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg" event={"ID":"89d32cd9-4b79-46cc-b807-0e183522f514","Type":"ContainerDied","Data":"68912ae549b1c59fca44533500467c56c2d91cae3a012fc7fb371ccdb5e75c08"} Nov 03 07:33:34.847114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.847091 4925 generic.go:356] "Generic (PLEG): container finished" podID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerID="bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818" exitCode=0 Nov 03 07:33:34.847217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.847122 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" event={"ID":"004ae909-4d9d-4755-be60-cc0b651ebdcd","Type":"ContainerDied","Data":"bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818"} Nov 03 07:33:34.847217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.847155 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" Nov 03 07:33:34.847217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.847180 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr" event={"ID":"004ae909-4d9d-4755-be60-cc0b651ebdcd","Type":"ContainerDied","Data":"323f258e124a412746c17ec9a194fa2ebb186d5f5a4641f2ccb2a994614a720d"} Nov 03 07:33:34.856101 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.856084 4925 scope.go:122] "RemoveContainer" containerID="4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905" Nov 03 07:33:34.856576 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:33:34.856555 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905\": container with ID starting with 4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905 not found: ID does not exist" containerID="4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905" Nov 03 07:33:34.856725 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.856692 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905"} err="failed to get container status \"4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905\": rpc error: code = NotFound desc = could not find container \"4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905\": container with ID starting with 4c22b3e040e32e6a57d0807d8404c3d6205a39c686ae45d297cfbaf9f9cb5905 not found: ID does not exist" Nov 03 07:33:34.856828 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.856816 4925 scope.go:122] "RemoveContainer" containerID="dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c" Nov 03 07:33:34.869002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.868982 4925 scope.go:122] "RemoveContainer" containerID="dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c" Nov 03 07:33:34.869414 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:33:34.869393 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c\": container with ID starting with dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c not found: ID does not exist" containerID="dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c" Nov 03 07:33:34.869489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.869423 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c"} err="failed to get container status \"dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c\": rpc error: code = NotFound desc = could not find container \"dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c\": container with ID starting with dfe7969422a8b6ce8776c659c0ae1d98c3c0ae77d87c88f3d512585298d0da2c not found: ID does not exist" Nov 03 07:33:34.869489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.869440 4925 scope.go:122] "RemoveContainer" containerID="bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818" Nov 03 07:33:34.878359 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.878340 4925 scope.go:122] "RemoveContainer" containerID="bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818" Nov 03 07:33:34.878626 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:33:34.878609 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818\": container with ID starting with bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818 not found: ID does not exist" containerID="bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818" Nov 03 07:33:34.878677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:34.878631 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818"} err="failed to get container status \"bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818\": rpc error: code = NotFound desc = could not find container \"bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818\": container with ID starting with bde13ac150bc990a7522aed589637352ed772e540cd284411961e8a04e481818 not found: ID does not exist" Nov 03 07:33:35.020502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:35.020469 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg"] Nov 03 07:33:35.026467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:35.026442 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg"] Nov 03 07:33:35.141025 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:35.140922 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr"] Nov 03 07:33:35.146978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:35.146953 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr"] Nov 03 07:33:35.261700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:35.261668 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb"] Nov 03 07:33:35.268077 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:35.268051 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb"] Nov 03 07:33:36.191810 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:36.191777 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" path="/var/lib/kubelet/pods/004ae909-4d9d-4755-be60-cc0b651ebdcd/volumes" Nov 03 07:33:36.192308 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:36.192292 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" path="/var/lib/kubelet/pods/7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd/volumes" Nov 03 07:33:36.192751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:36.192737 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" path="/var/lib/kubelet/pods/89d32cd9-4b79-46cc-b807-0e183522f514/volumes" Nov 03 07:33:39.139939 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:33:39.139888 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:33:39.640575 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:39.640542 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:39.829571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:39.829530 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:44.643870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:44.643827 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:49.646800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:49.646759 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:54.574038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:54.573300 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:33:57.605618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:57.605578 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:33:59.655994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:33:59.655957 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:01.740238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:01.740214 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:34:01.741621 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:01.741606 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:34:01.743707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:01.743686 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:34:01.751033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:01.745176 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:34:02.670502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:02.670458 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:05.245951 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:05.245815 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:07.473157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:07.473120 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:34:07.473574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:07.473182 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:34:09.140467 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:34:09.140429 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:34:09.149343 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:09.149315 4925 generic.go:356] "Generic (PLEG): container finished" podID="f1569787-fd4c-4255-aacf-5d74974813ac" containerID="a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a" exitCode=0 Nov 03 07:34:09.149489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:09.149368 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" event={"ID":"f1569787-fd4c-4255-aacf-5d74974813ac","Type":"ContainerDied","Data":"a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a"} Nov 03 07:34:09.149793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:09.149764 4925 scope.go:122] "RemoveContainer" containerID="a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a" Nov 03 07:34:09.151708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:09.151677 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:34:09.151786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:09.151753 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:34:09.151824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:09.151814 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:34:10.192778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:10.192742 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" event={"ID":"f1569787-fd4c-4255-aacf-5d74974813ac","Type":"ContainerStarted","Data":"ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316"} Nov 03 07:34:10.357553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:10.357461 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:34:10.358848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:10.358717 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 07:34:10.359091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:10.359064 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 07:34:10.727102 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:10.727067 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:11.371598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:11.371145 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 07:34:11.371598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:11.371212 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 07:34:12.357440 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:12.357405 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 07:34:12.357782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:12.357754 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 07:34:13.366036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:13.360397 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 07:34:13.366036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:13.360476 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 07:34:14.358451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:14.358406 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 07:34:14.358762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:14.358721 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 07:34:14.419391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:14.419357 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8"] Nov 03 07:34:14.420081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:14.420029 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" containerID="cri-o://54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c" gracePeriod=30 Nov 03 07:34:15.363206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:15.363171 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:34:15.363610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:15.363587 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:34:15.364243 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:15.364224 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:34:15.381401 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:15.381343 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:34:15.381587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:15.381434 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:34:15.677930 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:15.677841 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:17.510653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:17.510615 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:19.731474 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.731451 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:34:19.888507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.888460 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-envoy\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.888507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.888514 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-credential-socket\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.888799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.888559 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-kube-api-access-7ljr5\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-kube-api-access-7ljr5\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.888799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.888628 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-socket\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.888799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.888686 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istiod-ca-cert\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.888958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.888884 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-credential-socket" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.889083 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889058 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-token\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-token\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.889147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889113 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istiod-ca-cert" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:19.889392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889349 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-socket" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.889529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889512 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-data\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.889607 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889581 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-certs\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.889668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889627 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-podinfo\") pod \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\" (UID: \"15c152a1-8cbc-43c7-ba23-bb54f3ff2975\") " Nov 03 07:34:19.889791 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889765 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-data" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.889901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.889853 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-certs" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.890267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.890243 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.890366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.890270 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.890366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.890284 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.890366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.890297 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.890366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.890310 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.891378 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.891346 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-envoy" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.891683 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.891659 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-kube-api-access-7ljr5" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "kube-api-access-7ljr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.891842 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.891823 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-token" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.892160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.892137 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-podinfo" pod "15c152a1-8cbc-43c7-ba23-bb54f3ff2975" (UID: "15c152a1-8cbc-43c7-ba23-bb54f3ff2975"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:34:19.991659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.991563 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.991659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.991599 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljr5\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-kube-api-access-7ljr5\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.991659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.991615 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.991659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:19.991626 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c152a1-8cbc-43c7-ba23-bb54f3ff2975-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.274825 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.274731 4925 generic.go:356] "Generic (PLEG): container finished" podID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerID="54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c" exitCode=0 Nov 03 07:34:20.274825 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.274800 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" event={"ID":"15c152a1-8cbc-43c7-ba23-bb54f3ff2975","Type":"ContainerDied","Data":"54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c"} Nov 03 07:34:20.275093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.274866 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" event={"ID":"15c152a1-8cbc-43c7-ba23-bb54f3ff2975","Type":"ContainerDied","Data":"88e4449949cda8ae6a9b07be4c18bb860ef9149bb564ba3c7b482ccfdfebf194"} Nov 03 07:34:20.275093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.274904 4925 scope.go:122] "RemoveContainer" containerID="54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c" Nov 03 07:34:20.275572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.275552 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8" Nov 03 07:34:20.304888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.304857 4925 scope.go:122] "RemoveContainer" containerID="54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c" Nov 03 07:34:20.308718 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:34:20.308675 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c\": container with ID starting with 54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c not found: ID does not exist" containerID="54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c" Nov 03 07:34:20.308854 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.308728 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c"} err="failed to get container status \"54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c\": rpc error: code = NotFound desc = could not find container \"54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c\": container with ID starting with 54728c19322621041fb012490159dc7ec205dc22eb1107b3d28b4ba9b965c07c not found: ID does not exist" Nov 03 07:34:20.452843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.452806 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8"] Nov 03 07:34:20.461129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:20.461094 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8"] Nov 03 07:34:21.926594 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:21.923056 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:22.195505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:22.195412 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" path="/var/lib/kubelet/pods/15c152a1-8cbc-43c7-ba23-bb54f3ff2975/volumes" Nov 03 07:34:25.467547 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:25.467510 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:25.705635 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:25.705595 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:27.303485 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:27.303436 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:30.383881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:30.383842 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:30.592668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:30.592632 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:31.663805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:31.663739 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:33.523595 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:33.518692 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:33.712331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:33.712286 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:33.947158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:33.947120 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:34.463372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:34.463333 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:35.303460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:35.303422 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:35.496460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:35.496421 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:38.699986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:38.699943 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:38.948087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:38.948047 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:39.139839 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:34:39.139804 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:34:41.607835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:41.607788 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:41.798889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:41.798851 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:42.078096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:42.058469 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:44.144170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:44.144132 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:44.338364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:44.338321 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:45.122731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:45.122694 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:45.349858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:45.349825 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:49.356970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:49.356935 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:49.637353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:49.637267 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:49.827048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:49.826977 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:50.010979 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:50.010892 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:51.870958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:51.870919 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:34:52.943986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:52.943952 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:53.190211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:53.190170 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:34:54.658691 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:34:54.658654 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:00.237544 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:00.237502 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:03.017924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:03.017885 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:05.020122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:05.020076 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:06.741107 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:06.741072 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:06.967792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:06.967736 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:07.176522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:07.176475 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:07.392516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:07.392479 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:09.138932 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:35:09.138896 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:35:11.388901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:11.388852 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:12.407868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:12.407823 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:15.282482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:15.282444 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:15.517486 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:15.517443 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:17.648827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:17.648791 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:17.948492 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:17.948349 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:20.387878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:20.387834 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:20.571625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:20.571579 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:22.244004 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:22.243965 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:24.023152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:24.023109 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:24.258870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:24.258835 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:24.460049 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:24.459997 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:25.119431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:25.119398 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:28.675171 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:28.675133 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:30.226736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:30.226699 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:30.472094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:30.472043 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:30.665521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:30.665487 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:31.176736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:31.176696 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:32.948311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:32.948274 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:33.207460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:33.207376 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:33.410686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:33.404687 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:35.871069 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:35.871032 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:38.485032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:38.484527 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:38.682220 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:38.682181 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:38.889676 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:38.889587 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:39.139761 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:35:39.139685 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:35:39.262678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:39.262644 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:35:40.478314 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:40.478226 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228"] Nov 03 07:35:40.478705 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:40.478467 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" podUID="7df9991f-9961-49f6-9e5d-253d7f83482e" containerName="maas-api" containerID="cri-o://248eb11b73ad4e4e299b049c0ba1a60b5eed6ca0ba088bb6a698d778ab3bf231" gracePeriod=30 Nov 03 07:35:40.630838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:40.630805 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:41.040288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.036817 4925 generic.go:356] "Generic (PLEG): container finished" podID="7df9991f-9961-49f6-9e5d-253d7f83482e" containerID="248eb11b73ad4e4e299b049c0ba1a60b5eed6ca0ba088bb6a698d778ab3bf231" exitCode=0 Nov 03 07:35:41.040288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.036919 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" event={"ID":"7df9991f-9961-49f6-9e5d-253d7f83482e","Type":"ContainerDied","Data":"248eb11b73ad4e4e299b049c0ba1a60b5eed6ca0ba088bb6a698d778ab3bf231"} Nov 03 07:35:41.067892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.067868 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:35:41.163608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.163579 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-metrics-certs\") pod \"7df9991f-9961-49f6-9e5d-253d7f83482e\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " Nov 03 07:35:41.163816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.163803 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls\") pod \"7df9991f-9961-49f6-9e5d-253d7f83482e\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " Nov 03 07:35:41.163872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.163836 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df9991f-9961-49f6-9e5d-253d7f83482e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df9991f-9961-49f6-9e5d-253d7f83482e-openshift-service-ca\") pod \"7df9991f-9961-49f6-9e5d-253d7f83482e\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " Nov 03 07:35:41.163926 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.163880 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df9991f-9961-49f6-9e5d-253d7f83482e-kube-api-access-hjg9l\" (UniqueName: \"kubernetes.io/projected/7df9991f-9961-49f6-9e5d-253d7f83482e-kube-api-access-hjg9l\") pod \"7df9991f-9961-49f6-9e5d-253d7f83482e\" (UID: \"7df9991f-9961-49f6-9e5d-253d7f83482e\") " Nov 03 07:35:41.164215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.164186 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df9991f-9961-49f6-9e5d-253d7f83482e-openshift-service-ca" pod "7df9991f-9961-49f6-9e5d-253d7f83482e" (UID: "7df9991f-9961-49f6-9e5d-253d7f83482e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:41.164414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.164396 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df9991f-9961-49f6-9e5d-253d7f83482e-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:41.166062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.166031 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls" pod "7df9991f-9961-49f6-9e5d-253d7f83482e" (UID: "7df9991f-9961-49f6-9e5d-253d7f83482e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:41.166183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.166114 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-metrics-certs" pod "7df9991f-9961-49f6-9e5d-253d7f83482e" (UID: "7df9991f-9961-49f6-9e5d-253d7f83482e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:41.166259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.166243 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df9991f-9961-49f6-9e5d-253d7f83482e-kube-api-access-hjg9l" pod "7df9991f-9961-49f6-9e5d-253d7f83482e" (UID: "7df9991f-9961-49f6-9e5d-253d7f83482e"). InnerVolumeSpecName "kube-api-access-hjg9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:41.266171 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.266119 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:41.266171 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.266170 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjg9l\" (UniqueName: \"kubernetes.io/projected/7df9991f-9961-49f6-9e5d-253d7f83482e-kube-api-access-hjg9l\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:41.266376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:41.266187 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9991f-9961-49f6-9e5d-253d7f83482e-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:42.045076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.045045 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" Nov 03 07:35:42.045506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.045048 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228" event={"ID":"7df9991f-9961-49f6-9e5d-253d7f83482e","Type":"ContainerDied","Data":"e02743125fb204241a86ae8c431508710265a782de5ca7205211d31b6755551c"} Nov 03 07:35:42.045506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.045177 4925 scope.go:122] "RemoveContainer" containerID="248eb11b73ad4e4e299b049c0ba1a60b5eed6ca0ba088bb6a698d778ab3bf231" Nov 03 07:35:42.220475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.220441 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228"] Nov 03 07:35:42.264274 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.264240 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:42.271068 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.271045 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-09fd61-55f9f479bb-c5228"] Nov 03 07:35:42.531625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.531598 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:42.716516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:42.716473 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:43.263546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:43.263511 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:44.191738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:44.191705 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df9991f-9961-49f6-9e5d-253d7f83482e" path="/var/lib/kubelet/pods/7df9991f-9961-49f6-9e5d-253d7f83482e/volumes" Nov 03 07:35:46.677275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:46.677236 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:46.880026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:46.879984 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:47.960114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:47.960075 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:50.424973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:50.424937 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:50.684545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:50.684408 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:50.897554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:50.897518 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:51.664056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:51.663982 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:51.905635 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:51.905582 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:52.566853 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:52.566816 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4"] Nov 03 07:35:52.567115 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:52.567084 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" podUID="7fc9b45c-1517-47f2-959d-9c7c59069398" containerName="maas-api" containerID="cri-o://fee7f056cde181231666e3320d4b605475b29ef3a75cfeb42be2ae759c4e532a" gracePeriod=30 Nov 03 07:35:52.710865 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:52.710833 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:53.220444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.220206 4925 generic.go:356] "Generic (PLEG): container finished" podID="7fc9b45c-1517-47f2-959d-9c7c59069398" containerID="fee7f056cde181231666e3320d4b605475b29ef3a75cfeb42be2ae759c4e532a" exitCode=0 Nov 03 07:35:53.220444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.220272 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" event={"ID":"7fc9b45c-1517-47f2-959d-9c7c59069398","Type":"ContainerDied","Data":"fee7f056cde181231666e3320d4b605475b29ef3a75cfeb42be2ae759c4e532a"} Nov 03 07:35:53.220444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.220318 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" event={"ID":"7fc9b45c-1517-47f2-959d-9c7c59069398","Type":"ContainerDied","Data":"293fea73cf687a85b218ce1b5de0d2629ef40464b542ca05019bb1dabe2f178d"} Nov 03 07:35:53.220444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.220346 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="293fea73cf687a85b218ce1b5de0d2629ef40464b542ca05019bb1dabe2f178d" Nov 03 07:35:53.227983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.227956 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:35:53.273617 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.273583 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc9b45c-1517-47f2-959d-9c7c59069398-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc9b45c-1517-47f2-959d-9c7c59069398-openshift-service-ca\") pod \"7fc9b45c-1517-47f2-959d-9c7c59069398\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " Nov 03 07:35:53.273801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.273712 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-metrics-certs\") pod \"7fc9b45c-1517-47f2-959d-9c7c59069398\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " Nov 03 07:35:53.273801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.273781 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc9b45c-1517-47f2-959d-9c7c59069398-kube-api-access-9crkg\" (UniqueName: \"kubernetes.io/projected/7fc9b45c-1517-47f2-959d-9c7c59069398-kube-api-access-9crkg\") pod \"7fc9b45c-1517-47f2-959d-9c7c59069398\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " Nov 03 07:35:53.273879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.273866 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls\") pod \"7fc9b45c-1517-47f2-959d-9c7c59069398\" (UID: \"7fc9b45c-1517-47f2-959d-9c7c59069398\") " Nov 03 07:35:53.274056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.274025 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc9b45c-1517-47f2-959d-9c7c59069398-openshift-service-ca" pod "7fc9b45c-1517-47f2-959d-9c7c59069398" (UID: "7fc9b45c-1517-47f2-959d-9c7c59069398"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:53.274384 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.274363 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc9b45c-1517-47f2-959d-9c7c59069398-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:53.276411 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.276382 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-metrics-certs" pod "7fc9b45c-1517-47f2-959d-9c7c59069398" (UID: "7fc9b45c-1517-47f2-959d-9c7c59069398"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:53.276682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.276661 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc9b45c-1517-47f2-959d-9c7c59069398-kube-api-access-9crkg" pod "7fc9b45c-1517-47f2-959d-9c7c59069398" (UID: "7fc9b45c-1517-47f2-959d-9c7c59069398"). InnerVolumeSpecName "kube-api-access-9crkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:53.276798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.276766 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls" pod "7fc9b45c-1517-47f2-959d-9c7c59069398" (UID: "7fc9b45c-1517-47f2-959d-9c7c59069398"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:53.376275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.376236 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:53.376275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.376276 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9crkg\" (UniqueName: \"kubernetes.io/projected/7fc9b45c-1517-47f2-959d-9c7c59069398-kube-api-access-9crkg\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:53.376567 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:53.376320 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc9b45c-1517-47f2-959d-9c7c59069398-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:54.226119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:54.226089 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4" Nov 03 07:35:54.399526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:54.399493 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4"] Nov 03 07:35:54.452348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:54.452262 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72a642-7d74bd68-6h8j4"] Nov 03 07:35:54.866958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:54.866873 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:55.070707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:55.070665 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:55.330656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:55.329688 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:55.991643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:55.991607 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:35:56.192350 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:35:56.192322 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc9b45c-1517-47f2-959d-9c7c59069398" path="/var/lib/kubelet/pods/7fc9b45c-1517-47f2-959d-9c7c59069398/volumes" Nov 03 07:36:00.235255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:00.235171 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:03.352343 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:03.352299 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:03.619940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:03.619757 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:03.820106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:03.820067 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:04.055819 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:04.055741 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:04.253542 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:04.253499 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:05.336453 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:05.336407 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:09.146980 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:36:09.146936 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:36:10.608943 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:10.608895 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:10.874344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:10.874264 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:14.857752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:14.857704 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:15.058967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:15.058927 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:15.932457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:15.932392 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:16.777074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:16.777034 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:16.967511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:16.967477 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:18.669583 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:18.669407 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:18.858474 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:18.858436 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:20.941310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:20.941276 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:21.195872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:21.192068 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:21.384153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:21.384110 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:21.661880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:21.661846 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:21.864678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:21.864640 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:22.958263 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:22.958230 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:36:26.117781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:26.117745 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:30.910448 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:30.910397 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:31.106969 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:31.106934 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:36.054142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:36.054102 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:39.139102 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:36:39.139064 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:36:41.092575 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:41.092542 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:43.765961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:43.765922 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:45.416479 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:45.416449 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:47.195053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:47.194882 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:47.386412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:47.386377 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:47.572608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:47.572572 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:48.209252 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:48.209223 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:51.673473 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:51.673437 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb"] Nov 03 07:36:51.674218 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:51.674180 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" podUID="288b91f6-ab6a-4cba-a40c-b34bc396bf49" containerName="maas-api" containerID="cri-o://fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d" gracePeriod=30 Nov 03 07:36:51.830857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:51.830821 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:52.038668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.038625 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:36:52.142674 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.142633 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288b91f6-ab6a-4cba-a40c-b34bc396bf49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288b91f6-ab6a-4cba-a40c-b34bc396bf49-openshift-service-ca\") pod \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " Nov 03 07:36:52.147743 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.147261 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288b91f6-ab6a-4cba-a40c-b34bc396bf49-openshift-service-ca" pod "288b91f6-ab6a-4cba-a40c-b34bc396bf49" (UID: "288b91f6-ab6a-4cba-a40c-b34bc396bf49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:36:52.147927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.147732 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls\") pod \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " Nov 03 07:36:52.157038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.148157 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-metrics-certs\") pod \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " Nov 03 07:36:52.166846 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.158133 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288b91f6-ab6a-4cba-a40c-b34bc396bf49-kube-api-access-jrq6m\" (UniqueName: \"kubernetes.io/projected/288b91f6-ab6a-4cba-a40c-b34bc396bf49-kube-api-access-jrq6m\") pod \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\" (UID: \"288b91f6-ab6a-4cba-a40c-b34bc396bf49\") " Nov 03 07:36:52.167759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.167726 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288b91f6-ab6a-4cba-a40c-b34bc396bf49-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:52.170758 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.170471 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls" pod "288b91f6-ab6a-4cba-a40c-b34bc396bf49" (UID: "288b91f6-ab6a-4cba-a40c-b34bc396bf49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:36:52.234105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.211549 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288b91f6-ab6a-4cba-a40c-b34bc396bf49-kube-api-access-jrq6m" pod "288b91f6-ab6a-4cba-a40c-b34bc396bf49" (UID: "288b91f6-ab6a-4cba-a40c-b34bc396bf49"). InnerVolumeSpecName "kube-api-access-jrq6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:36:52.234105 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.212118 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-metrics-certs" pod "288b91f6-ab6a-4cba-a40c-b34bc396bf49" (UID: "288b91f6-ab6a-4cba-a40c-b34bc396bf49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:36:52.269452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.269342 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:52.269452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.269378 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288b91f6-ab6a-4cba-a40c-b34bc396bf49-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:52.269452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.269394 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrq6m\" (UniqueName: \"kubernetes.io/projected/288b91f6-ab6a-4cba-a40c-b34bc396bf49-kube-api-access-jrq6m\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:52.820734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.820698 4925 generic.go:356] "Generic (PLEG): container finished" podID="288b91f6-ab6a-4cba-a40c-b34bc396bf49" containerID="fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d" exitCode=0 Nov 03 07:36:52.821292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.820940 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" event={"ID":"288b91f6-ab6a-4cba-a40c-b34bc396bf49","Type":"ContainerDied","Data":"fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d"} Nov 03 07:36:52.821292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.820991 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" event={"ID":"288b91f6-ab6a-4cba-a40c-b34bc396bf49","Type":"ContainerDied","Data":"c32e8ec1719438cbb67f22f4209c8fd8ac4bf88a18fa46f3ad7324124198e7fb"} Nov 03 07:36:52.821292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.821053 4925 scope.go:122] "RemoveContainer" containerID="fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d" Nov 03 07:36:52.821292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.821155 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb" Nov 03 07:36:52.836647 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.836623 4925 scope.go:122] "RemoveContainer" containerID="fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d" Nov 03 07:36:52.837002 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:36:52.836973 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d\": container with ID starting with fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d not found: ID does not exist" containerID="fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d" Nov 03 07:36:52.837149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.837036 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d"} err="failed to get container status \"fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d\": rpc error: code = NotFound desc = could not find container \"fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d\": container with ID starting with fa04ebc9d056352761092728ace393049353b20c53932f95b0f836549b80e82d not found: ID does not exist" Nov 03 07:36:52.994556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:52.994475 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb"] Nov 03 07:36:53.045039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:53.044983 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-32b500-54547db75-2zpmb"] Nov 03 07:36:53.497087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:53.497058 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:53.783961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:53.783864 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:53.971921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:53.971882 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:54.194221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:54.194182 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288b91f6-ab6a-4cba-a40c-b34bc396bf49" path="/var/lib/kubelet/pods/288b91f6-ab6a-4cba-a40c-b34bc396bf49/volumes" Nov 03 07:36:54.410435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:54.410395 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:56.071825 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:56.065796 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:56.254546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:56.254510 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:56.463091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:56.462337 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:58.854640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:58.854602 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:36:59.554185 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:59.554150 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs"] Nov 03 07:36:59.554420 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:59.554381 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" podUID="9b9595de-7246-4ec1-8b88-b2286d081ebb" containerName="discovery" containerID="cri-o://eb87aa081b8348d6045893cdaef5a587fdf2b3e41ff823fbd47b1bd504746393" gracePeriod=30 Nov 03 07:36:59.912752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:59.912268 4925 generic.go:356] "Generic (PLEG): container finished" podID="9b9595de-7246-4ec1-8b88-b2286d081ebb" containerID="eb87aa081b8348d6045893cdaef5a587fdf2b3e41ff823fbd47b1bd504746393" exitCode=0 Nov 03 07:36:59.912752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:59.912397 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" event={"ID":"9b9595de-7246-4ec1-8b88-b2286d081ebb","Type":"ContainerDied","Data":"eb87aa081b8348d6045893cdaef5a587fdf2b3e41ff823fbd47b1bd504746393"} Nov 03 07:36:59.947417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:36:59.947392 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:37:00.048375 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.048340 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-kube-api-access-rvn5w\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-kube-api-access-rvn5w\") pod \"9b9595de-7246-4ec1-8b88-b2286d081ebb\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " Nov 03 07:37:00.048825 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.048792 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-dns-cert\") pod \"9b9595de-7246-4ec1-8b88-b2286d081ebb\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " Nov 03 07:37:00.048993 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.048977 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-ca-configmap\") pod \"9b9595de-7246-4ec1-8b88-b2286d081ebb\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " Nov 03 07:37:00.049245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.049228 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9595de-7246-4ec1-8b88-b2286d081ebb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9595de-7246-4ec1-8b88-b2286d081ebb-local-certs\") pod \"9b9595de-7246-4ec1-8b88-b2286d081ebb\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " Nov 03 07:37:00.049816 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.049796 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-kubeconfig\") pod \"9b9595de-7246-4ec1-8b88-b2286d081ebb\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " Nov 03 07:37:00.050325 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.049685 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-ca-configmap" pod "9b9595de-7246-4ec1-8b88-b2286d081ebb" (UID: "9b9595de-7246-4ec1-8b88-b2286d081ebb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:00.050461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.050268 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-cacerts\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-cacerts\") pod \"9b9595de-7246-4ec1-8b88-b2286d081ebb\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " Nov 03 07:37:00.051711 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.051550 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-kube-api-access-rvn5w" pod "9b9595de-7246-4ec1-8b88-b2286d081ebb" (UID: "9b9595de-7246-4ec1-8b88-b2286d081ebb"). InnerVolumeSpecName "kube-api-access-rvn5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:00.051873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.051844 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-token\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-token\") pod \"9b9595de-7246-4ec1-8b88-b2286d081ebb\" (UID: \"9b9595de-7246-4ec1-8b88-b2286d081ebb\") " Nov 03 07:37:00.053740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.053114 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvn5w\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-kube-api-access-rvn5w\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:00.053740 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.053144 4925 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-ca-configmap\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:00.054151 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.054116 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9595de-7246-4ec1-8b88-b2286d081ebb-local-certs" pod "9b9595de-7246-4ec1-8b88-b2286d081ebb" (UID: "9b9595de-7246-4ec1-8b88-b2286d081ebb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:00.054522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.054497 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-kubeconfig" pod "9b9595de-7246-4ec1-8b88-b2286d081ebb" (UID: "9b9595de-7246-4ec1-8b88-b2286d081ebb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:00.055629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.055606 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-token" pod "9b9595de-7246-4ec1-8b88-b2286d081ebb" (UID: "9b9595de-7246-4ec1-8b88-b2286d081ebb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:00.055733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.055686 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-cacerts" pod "9b9595de-7246-4ec1-8b88-b2286d081ebb" (UID: "9b9595de-7246-4ec1-8b88-b2286d081ebb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:00.055921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.055896 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-dns-cert" pod "9b9595de-7246-4ec1-8b88-b2286d081ebb" (UID: "9b9595de-7246-4ec1-8b88-b2286d081ebb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:00.154533 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.154493 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:00.154533 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.154527 4925 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-csr-dns-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:00.154533 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.154537 4925 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9595de-7246-4ec1-8b88-b2286d081ebb-local-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:00.154827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.154546 4925 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-istio-kubeconfig\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:00.154827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.154554 4925 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b9595de-7246-4ec1-8b88-b2286d081ebb-cacerts\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:00.529509 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.529431 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:00.814798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.814719 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:00.920109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.920071 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" event={"ID":"9b9595de-7246-4ec1-8b88-b2286d081ebb","Type":"ContainerDied","Data":"9532c2a7e9c4ca27deb0c80695f7b401ae167d52dd977969baf135043bed2a28"} Nov 03 07:37:00.920109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.920090 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs" Nov 03 07:37:00.920578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:00.920167 4925 scope.go:122] "RemoveContainer" containerID="eb87aa081b8348d6045893cdaef5a587fdf2b3e41ff823fbd47b1bd504746393" Nov 03 07:37:01.015183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:01.015149 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:01.144657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:01.144571 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs"] Nov 03 07:37:01.195333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:01.195298 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs"] Nov 03 07:37:01.513550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:01.513452 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:01.947577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:01.947544 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:02.137337 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:02.137292 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:02.192904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:02.192870 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9595de-7246-4ec1-8b88-b2286d081ebb" path="/var/lib/kubelet/pods/9b9595de-7246-4ec1-8b88-b2286d081ebb/volumes" Nov 03 07:37:02.260681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:02.260645 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks"] Nov 03 07:37:02.260968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:02.260939 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" containerID="cri-o://3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab" gracePeriod=30 Nov 03 07:37:02.323641 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:02.323606 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:03.390527 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:03.390492 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:03.626696 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:03.626657 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:03.816040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:03.815994 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:05.105517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:05.105485 4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:37:05.105901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:05.105546 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:37:05.656193 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:05.656153 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:06.928836 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:06.928792 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:07.414963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.414925 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:07.581404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.581381 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:37:07.682052 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682025 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-kube-api-access-p8dgz\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-kube-api-access-p8dgz\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682065 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210fc062-7755-456f-80d0-c2dc5366b0f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210fc062-7755-456f-80d0-c2dc5366b0f9-istiod-ca-cert\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682130 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-data\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682160 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-token\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-token\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682212 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-podinfo\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682238 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-envoy\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682293 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-socket\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682388 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-credential-socket\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682439 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-certs\") pod \"210fc062-7755-456f-80d0-c2dc5366b0f9\" (UID: \"210fc062-7755-456f-80d0-c2dc5366b0f9\") " Nov 03 07:37:07.682502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682465 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-data" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:07.682845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682556 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210fc062-7755-456f-80d0-c2dc5366b0f9-istiod-ca-cert" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:07.682845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.682694 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-socket" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:07.683110 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.683092 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.683184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.683123 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.683184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.683138 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210fc062-7755-456f-80d0-c2dc5366b0f9-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.683184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.683088 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-certs" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:07.683184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.683107 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-credential-socket" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:07.684652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.684617 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-podinfo" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:07.684652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.684620 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-token" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:07.685075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.685049 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-kube-api-access-p8dgz" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "kube-api-access-p8dgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:07.685173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.685090 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-envoy" pod "210fc062-7755-456f-80d0-c2dc5366b0f9" (UID: "210fc062-7755-456f-80d0-c2dc5366b0f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:07.751839 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.751808 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:07.785173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.785139 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.785173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.785169 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8dgz\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-kube-api-access-p8dgz\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.785365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.785185 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.785365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.785195 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.785365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.785203 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.785365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.785211 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210fc062-7755-456f-80d0-c2dc5366b0f9-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.976714 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.976678 4925 generic.go:356] "Generic (PLEG): container finished" podID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerID="3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab" exitCode=0 Nov 03 07:37:07.977189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.976735 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" Nov 03 07:37:07.977189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.976788 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" event={"ID":"210fc062-7755-456f-80d0-c2dc5366b0f9","Type":"ContainerDied","Data":"3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab"} Nov 03 07:37:07.977189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.976850 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks" event={"ID":"210fc062-7755-456f-80d0-c2dc5366b0f9","Type":"ContainerDied","Data":"feb168b99e5022afa551c01e1bf85396f9f3f7bb26cdca1d73821072928094d1"} Nov 03 07:37:07.977189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.976878 4925 scope.go:122] "RemoveContainer" containerID="3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab" Nov 03 07:37:07.987866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.987848 4925 scope.go:122] "RemoveContainer" containerID="2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb" Nov 03 07:37:07.997321 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.997296 4925 scope.go:122] "RemoveContainer" containerID="3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab" Nov 03 07:37:07.997566 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:07.997547 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab\": container with ID starting with 3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab not found: ID does not exist" containerID="3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab" Nov 03 07:37:07.997634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.997576 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab"} err="failed to get container status \"3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab\": rpc error: code = NotFound desc = could not find container \"3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab\": container with ID starting with 3c37b572c2e38eb66eaf7b30b081dd51661519940df1b49cebf3444223f150ab not found: ID does not exist" Nov 03 07:37:07.997634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.997594 4925 scope.go:122] "RemoveContainer" containerID="2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb" Nov 03 07:37:07.997823 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:07.997806 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb\": container with ID starting with 2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb not found: ID does not exist" containerID="2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb" Nov 03 07:37:07.997867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:07.997828 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb"} err="failed to get container status \"2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb\": rpc error: code = NotFound desc = could not find container \"2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb\": container with ID starting with 2770b4e60c2dafead81612d3bf371e8ed57553abf1efc2bbf93cecf09b8c0ccb not found: ID does not exist" Nov 03 07:37:08.150322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:08.150287 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks"] Nov 03 07:37:08.158983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:08.158943 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks"] Nov 03 07:37:08.195324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:08.195284 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" path="/var/lib/kubelet/pods/210fc062-7755-456f-80d0-c2dc5366b0f9/volumes" Nov 03 07:37:08.379392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:08.379355 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:08.859659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:08.859621 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:09.050781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:09.050746 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:09.165112 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:09.151116 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:37:09.238034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:09.237978 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:09.540715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:09.540621 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:09.743536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:09.743503 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:09.878322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:09.878288 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr"] Nov 03 07:37:09.878558 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:09.878533 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" containerID="cri-o://4eb36bb8fc296e9b6925b3735ec38db1ff86f003dcec40625b14121e4747fa18" gracePeriod=30 Nov 03 07:37:09.940994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:09.940953 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:10.181587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:10.181489 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:10.380891 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:10.380808 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:10.815358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:10.815314 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:11.432438 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:11.432392 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:11.718311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:11.718209 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:14.397624 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:14.397573 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:14.640758 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:14.640720 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:14.841732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:14.841696 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:15.032297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.032261 4925 generic.go:356] "Generic (PLEG): container finished" podID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerID="4eb36bb8fc296e9b6925b3735ec38db1ff86f003dcec40625b14121e4747fa18" exitCode=0 Nov 03 07:37:15.032466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.032333 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" event={"ID":"a1e200ac-14ee-4353-8101-31c908fa3dcd","Type":"ContainerDied","Data":"4eb36bb8fc296e9b6925b3735ec38db1ff86f003dcec40625b14121e4747fa18"} Nov 03 07:37:15.032466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.032407 4925 scope.go:122] "RemoveContainer" containerID="72288a1bf896f3839b44379bc0c51b29f54a418ab446c210f96d96367159b759" Nov 03 07:37:15.184423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.183345 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:37:15.322400 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322362 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-credential-socket\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.322572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322436 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-kube-api-access-whth6\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-kube-api-access-whth6\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.322572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322492 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-data\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.322572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322533 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-socket\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.322935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322624 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-envoy\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.322935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322669 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-credential-socket" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:15.322935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322745 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e200ac-14ee-4353-8101-31c908fa3dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e200ac-14ee-4353-8101-31c908fa3dcd-istiod-ca-cert\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.323160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.322994 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-podinfo\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.323160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323063 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-certs\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.323160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323099 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-token\") pod \"a1e200ac-14ee-4353-8101-31c908fa3dcd\" (UID: \"a1e200ac-14ee-4353-8101-31c908fa3dcd\") " Nov 03 07:37:15.323160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323111 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-socket" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:15.323374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323247 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e200ac-14ee-4353-8101-31c908fa3dcd-istiod-ca-cert" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:15.323436 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323379 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-certs" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:15.323613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323569 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-data" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:15.323845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323822 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.323928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323853 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.323928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323870 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e200ac-14ee-4353-8101-31c908fa3dcd-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.323928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323882 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.323928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.323891 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.325619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.325594 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-envoy" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:15.325719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.325685 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-token" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:15.325719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.325703 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-podinfo" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:15.325719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.325714 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-kube-api-access-whth6" pod "a1e200ac-14ee-4353-8101-31c908fa3dcd" (UID: "a1e200ac-14ee-4353-8101-31c908fa3dcd"). InnerVolumeSpecName "kube-api-access-whth6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:15.425427 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.425351 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whth6\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-kube-api-access-whth6\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.425427 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.425378 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.425427 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.425388 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.425427 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.425399 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e200ac-14ee-4353-8101-31c908fa3dcd-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.468377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.468344 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:15.716054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.715791 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:15.764812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.764765 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8"] Nov 03 07:37:15.765501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.765062 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" containerID="cri-o://ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316" gracePeriod=30 Nov 03 07:37:15.918502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:15.918468 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:16.040580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:16.040499 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" Nov 03 07:37:16.040580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:16.040515 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr" event={"ID":"a1e200ac-14ee-4353-8101-31c908fa3dcd","Type":"ContainerDied","Data":"62dcbc04004af03dfb200442de4835b0337a959b2d738fc278adcfac55164d06"} Nov 03 07:37:16.040580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:16.040570 4925 scope.go:122] "RemoveContainer" containerID="4eb36bb8fc296e9b6925b3735ec38db1ff86f003dcec40625b14121e4747fa18" Nov 03 07:37:16.117715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:16.117672 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:16.216299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:16.216274 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr"] Nov 03 07:37:16.219416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:16.219395 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr"] Nov 03 07:37:16.649663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:16.649613 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:17.754472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:17.754441 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:17.938229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:17.938191 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:18.151258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:18.151216 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:18.199246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:18.199206 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" path="/var/lib/kubelet/pods/a1e200ac-14ee-4353-8101-31c908fa3dcd/volumes" Nov 03 07:37:18.578637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:18.577820 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:18.779610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:18.779565 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:19.511279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:19.511248 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:19.750736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:19.750684 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:20.008714 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:20.008682 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:20.200849 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:20.200815 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:20.821197 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:20.820414 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:21.083098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.083067 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:37:21.092737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.092706 4925 generic.go:356] "Generic (PLEG): container finished" podID="f1569787-fd4c-4255-aacf-5d74974813ac" containerID="ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316" exitCode=0 Nov 03 07:37:21.092895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.092762 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" Nov 03 07:37:21.092895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.092799 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" event={"ID":"f1569787-fd4c-4255-aacf-5d74974813ac","Type":"ContainerDied","Data":"ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316"} Nov 03 07:37:21.092895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.092862 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8" event={"ID":"f1569787-fd4c-4255-aacf-5d74974813ac","Type":"ContainerDied","Data":"fe10d8f2820743d3865de04d471581ae6d9c2a3bcae96bbf37f08b130ccebe64"} Nov 03 07:37:21.093113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.092895 4925 scope.go:122] "RemoveContainer" containerID="ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316" Nov 03 07:37:21.104026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.103990 4925 scope.go:122] "RemoveContainer" containerID="a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a" Nov 03 07:37:21.113428 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.113396 4925 scope.go:122] "RemoveContainer" containerID="ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316" Nov 03 07:37:21.113679 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:21.113658 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316\": container with ID starting with ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316 not found: ID does not exist" containerID="ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316" Nov 03 07:37:21.113748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.113686 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316"} err="failed to get container status \"ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316\": rpc error: code = NotFound desc = could not find container \"ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316\": container with ID starting with ffa52eff6d5ef26393d0320b0a88822022a56a752a09c5be63728dff963ba316 not found: ID does not exist" Nov 03 07:37:21.113748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.113703 4925 scope.go:122] "RemoveContainer" containerID="a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a" Nov 03 07:37:21.113944 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:21.113924 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a\": container with ID starting with a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a not found: ID does not exist" containerID="a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a" Nov 03 07:37:21.114043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.113950 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a"} err="failed to get container status \"a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a\": rpc error: code = NotFound desc = could not find container \"a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a\": container with ID starting with a70dfe689c8cbe5581e88225c5c199aaf9bbee8bd90e11c2a24924b0dcb69d4a not found: ID does not exist" Nov 03 07:37:21.124752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.124731 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-socket\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.124889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.124765 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1569787-fd4c-4255-aacf-5d74974813ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1569787-fd4c-4255-aacf-5d74974813ac-istio-podinfo\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.124889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.124804 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-istio-token\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-istio-token\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.124889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.124853 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-envoy\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.125080 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125051 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-socket" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:21.125080 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125065 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-certs\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.125190 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125119 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1569787-fd4c-4255-aacf-5d74974813ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1569787-fd4c-4255-aacf-5d74974813ac-istiod-ca-cert\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.125244 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125194 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-data\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.125300 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125254 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-kube-api-access-xj9b7\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-kube-api-access-xj9b7\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.125300 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125292 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-credential-socket\") pod \"f1569787-fd4c-4255-aacf-5d74974813ac\" (UID: \"f1569787-fd4c-4255-aacf-5d74974813ac\") " Nov 03 07:37:21.125410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125312 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-certs" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:21.125526 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125509 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-data" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:21.125789 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125726 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1569787-fd4c-4255-aacf-5d74974813ac-istiod-ca-cert" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:21.126025 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.125989 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-credential-socket" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:21.126242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.126108 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.126242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.126131 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.126242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.126147 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.126242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.126161 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1569787-fd4c-4255-aacf-5d74974813ac-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.127815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.127766 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-kube-api-access-xj9b7" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "kube-api-access-xj9b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:21.127915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.127872 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-envoy" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:21.128220 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.128198 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-istio-token" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:21.128293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.128257 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1569787-fd4c-4255-aacf-5d74974813ac-istio-podinfo" pod "f1569787-fd4c-4255-aacf-5d74974813ac" (UID: "f1569787-fd4c-4255-aacf-5d74974813ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:21.227661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.227392 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1569787-fd4c-4255-aacf-5d74974813ac-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.227661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.227427 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.227661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.227443 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.227661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.227459 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj9b7\" (UniqueName: \"kubernetes.io/projected/f1569787-fd4c-4255-aacf-5d74974813ac-kube-api-access-xj9b7\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.227661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.227475 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1569787-fd4c-4255-aacf-5d74974813ac-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:21.255778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.255671 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:21.524196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.524157 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8"] Nov 03 07:37:21.528107 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:21.528079 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8"] Nov 03 07:37:22.194085 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:22.194050 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" path="/var/lib/kubelet/pods/f1569787-fd4c-4255-aacf-5d74974813ac/volumes" Nov 03 07:37:22.944506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:22.944471 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:23.194961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.194875 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:23.362588 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.362553 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6"] Nov 03 07:37:23.362815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.362792 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" containerID="cri-o://e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714" gracePeriod=30 Nov 03 07:37:23.396677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.396630 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:23.413026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.412981 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8"] Nov 03 07:37:23.413280 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.413255 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" containerID="cri-o://abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27" gracePeriod=30 Nov 03 07:37:23.416847 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.416820 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr"] Nov 03 07:37:23.417152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.417107 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" containerID="cri-o://e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa" gracePeriod=30 Nov 03 07:37:23.464164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.464086 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn"] Nov 03 07:37:23.464349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.464310 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" containerID="cri-o://bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c" gracePeriod=30 Nov 03 07:37:23.713872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:23.713833 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:24.213181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213150 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4"] Nov 03 07:37:24.213696 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213655 4925 cpu_manager.go:414] "containerMap: removing container" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" Nov 03 07:37:24.213696 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213666 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" Nov 03 07:37:24.213696 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213675 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9b9595de-7246-4ec1-8b88-b2286d081ebb" containerName="discovery" Nov 03 07:37:24.213696 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213680 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9595de-7246-4ec1-8b88-b2286d081ebb" containerName="discovery" Nov 03 07:37:24.213696 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213689 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" Nov 03 07:37:24.213696 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213696 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213708 4925 cpu_manager.go:414] "containerMap: removing container" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213713 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213719 4925 cpu_manager.go:414] "containerMap: removing container" podUID="288b91f6-ab6a-4cba-a40c-b34bc396bf49" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213723 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288b91f6-ab6a-4cba-a40c-b34bc396bf49" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213733 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213737 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213747 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213751 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213758 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7fc9b45c-1517-47f2-959d-9c7c59069398" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213763 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc9b45c-1517-47f2-959d-9c7c59069398" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213771 4925 cpu_manager.go:414] "containerMap: removing container" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213775 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213787 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213792 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213797 4925 cpu_manager.go:414] "containerMap: removing container" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213802 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213810 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7df9991f-9961-49f6-9e5d-253d7f83482e" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213816 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df9991f-9961-49f6-9e5d-253d7f83482e" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213827 4925 cpu_manager.go:414] "containerMap: removing container" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213831 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213906 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c152a1-8cbc-43c7-ba23-bb54f3ff2975" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213916 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="004ae909-4d9d-4755-be60-cc0b651ebdcd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213921 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9595de-7246-4ec1-8b88-b2286d081ebb" containerName="discovery" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213928 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213934 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213939 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="288b91f6-ab6a-4cba-a40c-b34bc396bf49" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213946 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e200ac-14ee-4353-8101-31c908fa3dcd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213953 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213958 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc9b45c-1517-47f2-959d-9c7c59069398" containerName="maas-api" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213964 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213970 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5d0227-bbfb-4ff3-9b8f-2a57d4d838dd" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213975 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d32cd9-4b79-46cc-b807-0e183522f514" containerName="istio-proxy" Nov 03 07:37:24.214035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.213980 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df9991f-9961-49f6-9e5d-253d7f83482e" containerName="maas-api" Nov 03 07:37:24.217442 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.217417 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.268123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.268085 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-token\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.277963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.277933 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tds\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-kube-api-access-c9tds\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.297354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.297311 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-socket\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.298978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.298948 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-podinfo\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.299282 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.299260 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-certs\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.299447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.299428 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-credential-socket\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.299602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.299586 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-envoy\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.304204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.303062 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52ae06b-ed08-47b0-be5f-2e522283b659-istiod-ca-cert\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.304401 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.304380 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-data\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.313033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.278437 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dcdb23-openshift-default-dockercfg-pplj5\"" Nov 03 07:37:24.313417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.313390 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4"] Nov 03 07:37:24.323052 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.322889 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.406757 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-envoy\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.406804 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52ae06b-ed08-47b0-be5f-2e522283b659-istiod-ca-cert\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.406842 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-data\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.406938 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-token\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.406970 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tds\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-kube-api-access-c9tds\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.407105 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-socket\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.407152 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-podinfo\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.407237 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-certs\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.411032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.407272 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-credential-socket\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.416796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.416752 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-socket\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.417187 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.417086 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-credential-socket\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.418224 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.417988 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-certs\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.419078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.419055 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52ae06b-ed08-47b0-be5f-2e522283b659-istiod-ca-cert\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.419523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.419504 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-data\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.420640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.420618 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-podinfo\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.424482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.424456 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-envoy\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.481295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.481207 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-token\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.481637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.481609 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tds\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-kube-api-access-c9tds\") pod \"e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.515647 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.515610 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:24.531361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.531331 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:24.703268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.703226 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:24.996639 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:24.996602 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52ae06b_ed08_47b0_be5f_2e522283b659.slice/crio-ccf15ac01cf9b6ae08dacfadadf6c8091f73e6509e98850ef12e777e06e3c5ab WatchSource:0}: Error finding container ccf15ac01cf9b6ae08dacfadadf6c8091f73e6509e98850ef12e777e06e3c5ab: Status 404 returned error can't find the container with id ccf15ac01cf9b6ae08dacfadadf6c8091f73e6509e98850ef12e777e06e3c5ab Nov 03 07:37:24.997932 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.997901 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4"] Nov 03 07:37:24.999193 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.999166 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:24.999268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.999227 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:24.999268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:24.999259 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:25.131900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:25.131861 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" event={"ID":"a52ae06b-ed08-47b0-be5f-2e522283b659","Type":"ContainerStarted","Data":"0a9cae86b01ff5f8b9c989c760d7a0b9eb486aafe25d6a803538dc26c0f671ab"} Nov 03 07:37:25.132147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:25.131928 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" event={"ID":"a52ae06b-ed08-47b0-be5f-2e522283b659","Type":"ContainerStarted","Data":"ccf15ac01cf9b6ae08dacfadadf6c8091f73e6509e98850ef12e777e06e3c5ab"} Nov 03 07:37:25.261319 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:25.261199 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podStartSLOduration=1.261183014 podStartE2EDuration="1.261183014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:25.257769625 +0000 UTC m=+2305.191298282" watchObservedRunningTime="2026-11-03 07:37:25.261183014 +0000 UTC m=+2305.194711664" Nov 03 07:37:25.531599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:25.531510 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:25.671036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:25.665678 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:26.537834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:26.537798 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:26.623655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:26.623617 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:26.866349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:26.866318 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:27.146785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:27.146702 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:27.147761 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:27.147737 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:37:28.215666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.215631 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:37:28.216150 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.215690 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:37:28.705936 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.705909 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:37:28.768581 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768546 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-certs\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.768765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768636 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-socket\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.768765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768667 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9e5265-25ca-4578-9533-096c5818509d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e5265-25ca-4578-9533-096c5818509d-istiod-ca-cert\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.768765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768749 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-kube-api-access-7zncl\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-kube-api-access-7zncl\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.768960 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768801 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-credential-socket\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.768960 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768916 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-certs" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.768960 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768957 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-data\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.769358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.768995 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-istio-token\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-istio-token\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.769358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.769057 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da9e5265-25ca-4578-9533-096c5818509d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e5265-25ca-4578-9533-096c5818509d-istio-podinfo\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.769358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.769235 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-credential-socket" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.769685 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.769658 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-socket" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.770207 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.770026 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-data" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.770565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.770340 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9e5265-25ca-4578-9533-096c5818509d-istiod-ca-cert" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:28.773043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.772988 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-istio-token" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.780446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.780417 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.780536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.780450 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.780536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.780464 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.780536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.780480 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.780536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.780494 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9e5265-25ca-4578-9533-096c5818509d-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.780536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.780508 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.783638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.783611 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-kube-api-access-7zncl" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "kube-api-access-7zncl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.784273 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.784232 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da9e5265-25ca-4578-9533-096c5818509d-istio-podinfo" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:28.803961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.803938 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:37:28.839898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.839873 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:37:28.845545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.845519 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:37:28.881865 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.881811 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-credential-socket\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.882047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.881878 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-envoy\") pod \"da9e5265-25ca-4578-9533-096c5818509d\" (UID: \"da9e5265-25ca-4578-9533-096c5818509d\") " Nov 03 07:37:28.882047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.881935 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-credential-socket\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.882047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.881997 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istiod-ca-cert\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.882213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882053 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-kube-api-access-85r7d\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-kube-api-access-85r7d\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.882213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882081 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-podinfo\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.882213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882110 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e5e306e-987f-415d-994a-3d80a1673dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e306e-987f-415d-994a-3d80a1673dff-istio-podinfo\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.882213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882137 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-token\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-token\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.882213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882162 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-podinfo\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.882213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882193 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-envoy\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.882423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882225 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-envoy\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.882423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882290 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-data\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.882423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882318 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-certs\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.882423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882356 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-socket\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.882423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.882386 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-kube-api-access-kh7vn\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-kube-api-access-kh7vn\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883352 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-credential-socket" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883367 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-kube-api-access-5pt68\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-kube-api-access-5pt68\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883481 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-data" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883505 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-token\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-token\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883550 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-data\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883629 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5e306e-987f-415d-994a-3d80a1673dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e306e-987f-415d-994a-3d80a1673dff-istiod-ca-cert\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883668 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-certs\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883714 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-socket\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883755 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-certs\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883794 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-istio-token\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883825 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-socket\") pod \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\" (UID: \"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883872 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-data\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883913 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istiod-ca-cert\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.883954 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-credential-socket\") pod \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\" (UID: \"43682d6b-e6ca-4ea0-be2f-91dc22fe735c\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.884034 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-envoy\") pod \"7e5e306e-987f-415d-994a-3d80a1673dff\" (UID: \"7e5e306e-987f-415d-994a-3d80a1673dff\") " Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.884393 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-credential-socket" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.884579 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istiod-ca-cert" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.884751 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-certs" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.884976 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.885002 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9e5265-25ca-4578-9533-096c5818509d-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.885051 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.885066 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.885083 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zncl\" (UniqueName: \"kubernetes.io/projected/da9e5265-25ca-4578-9533-096c5818509d-kube-api-access-7zncl\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.885101 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.885322 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-certs" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.886177 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-socket" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.886472 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5e306e-987f-415d-994a-3d80a1673dff-istiod-ca-cert" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.886713 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-data" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.886921 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-data" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.887314 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-socket" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.887458 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istiod-ca-cert" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.887580 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-certs" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.887806 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-socket" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.887845 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-credential-socket" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.887863 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-envoy" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.888978 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.888937 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-istio-token" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.891164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.889885 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-podinfo" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:28.891164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.890321 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-envoy" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.891164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.890475 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-token" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.891164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.890603 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-envoy" pod "da9e5265-25ca-4578-9533-096c5818509d" (UID: "da9e5265-25ca-4578-9533-096c5818509d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.891164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.890703 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-podinfo" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:28.891164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.890825 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e5e306e-987f-415d-994a-3d80a1673dff-istio-podinfo" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:28.891164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.890959 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-envoy" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:28.892284 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.892256 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-token" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.892536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.892514 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-kube-api-access-kh7vn" pod "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" (UID: "6b08ccb5-c6f4-4f73-b65a-aa547191ebb0"). InnerVolumeSpecName "kube-api-access-kh7vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.892852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.892831 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-kube-api-access-5pt68" pod "43682d6b-e6ca-4ea0-be2f-91dc22fe735c" (UID: "43682d6b-e6ca-4ea0-be2f-91dc22fe735c"). InnerVolumeSpecName "kube-api-access-5pt68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.892988 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.892964 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-kube-api-access-85r7d" pod "7e5e306e-987f-415d-994a-3d80a1673dff" (UID: "7e5e306e-987f-415d-994a-3d80a1673dff"). InnerVolumeSpecName "kube-api-access-85r7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:28.986604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986525 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986562 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986574 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986583 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986596 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986606 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh7vn\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-kube-api-access-kh7vn\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986615 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pt68\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-kube-api-access-5pt68\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986624 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986632 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986641 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5e306e-987f-415d-994a-3d80a1673dff-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986649 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986658 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986666 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986675 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986684 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986692 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986700 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986711 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986721 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5e306e-987f-415d-994a-3d80a1673dff-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986730 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9e5265-25ca-4578-9533-096c5818509d-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986740 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85r7d\" (UniqueName: \"kubernetes.io/projected/7e5e306e-987f-415d-994a-3d80a1673dff-kube-api-access-85r7d\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986749 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986757 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5e306e-987f-415d-994a-3d80a1673dff-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:28.986929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:28.986765 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43682d6b-e6ca-4ea0-be2f-91dc22fe735c-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:29.164452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.164417 4925 generic.go:356] "Generic (PLEG): container finished" podID="da9e5265-25ca-4578-9533-096c5818509d" containerID="e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714" exitCode=0 Nov 03 07:37:29.164610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.164480 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" Nov 03 07:37:29.164610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.164568 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" event={"ID":"da9e5265-25ca-4578-9533-096c5818509d","Type":"ContainerDied","Data":"e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714"} Nov 03 07:37:29.164690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.164610 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6" event={"ID":"da9e5265-25ca-4578-9533-096c5818509d","Type":"ContainerDied","Data":"a395dae042e5e0b366545fc649f426f672fddc0de849f30fe9334e471730e771"} Nov 03 07:37:29.164690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.164638 4925 scope.go:122] "RemoveContainer" containerID="e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714" Nov 03 07:37:29.169171 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.169140 4925 generic.go:356] "Generic (PLEG): container finished" podID="7e5e306e-987f-415d-994a-3d80a1673dff" containerID="bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c" exitCode=0 Nov 03 07:37:29.169367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.169349 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" Nov 03 07:37:29.169853 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.169764 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" event={"ID":"7e5e306e-987f-415d-994a-3d80a1673dff","Type":"ContainerDied","Data":"bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c"} Nov 03 07:37:29.169853 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.169819 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn" event={"ID":"7e5e306e-987f-415d-994a-3d80a1673dff","Type":"ContainerDied","Data":"b69609eca5875b3a2ff2ef8ec8630b04b7b928f68e5493b67831063a727a2dbe"} Nov 03 07:37:29.171762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.171740 4925 generic.go:356] "Generic (PLEG): container finished" podID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerID="abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27" exitCode=0 Nov 03 07:37:29.175660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.172188 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" Nov 03 07:37:29.175660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.173828 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" event={"ID":"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0","Type":"ContainerDied","Data":"abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27"} Nov 03 07:37:29.175660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.173872 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" event={"ID":"6b08ccb5-c6f4-4f73-b65a-aa547191ebb0","Type":"ContainerDied","Data":"97a01ecfc06c2c7ab4cae800d80eb1e8624e0216174964cd663f8c7db37aae1c"} Nov 03 07:37:29.175660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.175358 4925 generic.go:356] "Generic (PLEG): container finished" podID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerID="e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa" exitCode=0 Nov 03 07:37:29.182086 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.175970 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" event={"ID":"43682d6b-e6ca-4ea0-be2f-91dc22fe735c","Type":"ContainerDied","Data":"e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa"} Nov 03 07:37:29.182086 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.176038 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" event={"ID":"43682d6b-e6ca-4ea0-be2f-91dc22fe735c","Type":"ContainerDied","Data":"c98b2a90389d2606be894619e8052a2dbf9e228ed51c461fc1d3f3c28fb02e3c"} Nov 03 07:37:29.183119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.183096 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr" Nov 03 07:37:29.183937 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.183922 4925 scope.go:122] "RemoveContainer" containerID="bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91" Nov 03 07:37:29.199904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.199885 4925 scope.go:122] "RemoveContainer" containerID="e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714" Nov 03 07:37:29.200232 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:29.200212 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714\": container with ID starting with e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714 not found: ID does not exist" containerID="e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714" Nov 03 07:37:29.200285 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.200241 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714"} err="failed to get container status \"e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714\": rpc error: code = NotFound desc = could not find container \"e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714\": container with ID starting with e74ec8579c9c6fc518606be252f04181ea9a539ff2f5af0bf013231607563714 not found: ID does not exist" Nov 03 07:37:29.200285 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.200258 4925 scope.go:122] "RemoveContainer" containerID="bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91" Nov 03 07:37:29.200534 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:29.200516 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91\": container with ID starting with bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91 not found: ID does not exist" containerID="bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91" Nov 03 07:37:29.200586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.200537 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91"} err="failed to get container status \"bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91\": rpc error: code = NotFound desc = could not find container \"bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91\": container with ID starting with bba4360e64e1c852ca9afb28093f264c972ae5aa2e2afa979e3d56746d0e7f91 not found: ID does not exist" Nov 03 07:37:29.200586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.200549 4925 scope.go:122] "RemoveContainer" containerID="bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c" Nov 03 07:37:29.210784 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.210760 4925 scope.go:122] "RemoveContainer" containerID="bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c" Nov 03 07:37:29.211173 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:29.211143 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c\": container with ID starting with bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c not found: ID does not exist" containerID="bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c" Nov 03 07:37:29.211268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.211182 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c"} err="failed to get container status \"bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c\": rpc error: code = NotFound desc = could not find container \"bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c\": container with ID starting with bea4b972d1ad7641cb277dadeef94f31d1a0bd1a93e91c840f1fcc4382bc1b3c not found: ID does not exist" Nov 03 07:37:29.211268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.211204 4925 scope.go:122] "RemoveContainer" containerID="abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27" Nov 03 07:37:29.220725 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.220696 4925 scope.go:122] "RemoveContainer" containerID="abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27" Nov 03 07:37:29.221072 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:29.221040 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27\": container with ID starting with abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27 not found: ID does not exist" containerID="abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27" Nov 03 07:37:29.221144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.221065 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27"} err="failed to get container status \"abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27\": rpc error: code = NotFound desc = could not find container \"abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27\": container with ID starting with abf9e7123d7915aa6a938377b38853a789d018ef4a1023f67455abdb12c8fd27 not found: ID does not exist" Nov 03 07:37:29.221144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.221081 4925 scope.go:122] "RemoveContainer" containerID="e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa" Nov 03 07:37:29.230394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.230378 4925 scope.go:122] "RemoveContainer" containerID="e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa" Nov 03 07:37:29.230662 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:29.230639 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa\": container with ID starting with e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa not found: ID does not exist" containerID="e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa" Nov 03 07:37:29.230752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.230665 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa"} err="failed to get container status \"e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa\": rpc error: code = NotFound desc = could not find container \"e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa\": container with ID starting with e1b3a27c6ac2d1d55e710713a3f4ae4ab17a1b7a03c73c849135887ed9bb98fa not found: ID does not exist" Nov 03 07:37:29.312290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.312190 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:29.458057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.456916 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6"] Nov 03 07:37:29.461066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.460987 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6"] Nov 03 07:37:29.501066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.500996 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:29.510215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.510187 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8"] Nov 03 07:37:29.516325 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.516287 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8"] Nov 03 07:37:29.632119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.632081 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn"] Nov 03 07:37:29.638354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.638326 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn"] Nov 03 07:37:29.695733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.695698 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54"] Nov 03 07:37:29.696228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696214 4925 cpu_manager.go:414] "containerMap: removing container" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" Nov 03 07:37:29.696299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696229 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" Nov 03 07:37:29.696299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696252 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" Nov 03 07:37:29.696299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696257 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" Nov 03 07:37:29.696299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696269 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" Nov 03 07:37:29.696299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696275 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" Nov 03 07:37:29.696299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696285 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" Nov 03 07:37:29.696299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696290 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1569787-fd4c-4255-aacf-5d74974813ac" containerName="istio-proxy" Nov 03 07:37:29.696519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696322 4925 cpu_manager.go:414] "containerMap: removing container" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" Nov 03 07:37:29.696519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696330 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" Nov 03 07:37:29.696519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696413 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" containerName="istio-proxy" Nov 03 07:37:29.696519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696421 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" Nov 03 07:37:29.696519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696430 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" containerName="istio-proxy" Nov 03 07:37:29.696519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696436 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="210fc062-7755-456f-80d0-c2dc5366b0f9" containerName="istio-proxy" Nov 03 07:37:29.696519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.696443 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" Nov 03 07:37:29.701274 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.701257 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:29.704271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.704239 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:37:29.704364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.704314 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:37:29.751280 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.751251 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr"] Nov 03 07:37:29.756601 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.756576 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1fb4bc\"" Nov 03 07:37:29.757099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.757079 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1fb4bc-dockercfg-dzr59\"" Nov 03 07:37:29.758821 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.758793 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54"] Nov 03 07:37:29.762191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.762165 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr"] Nov 03 07:37:29.810162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.810123 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl45z\" (UniqueName: \"kubernetes.io/projected/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-kube-api-access-vl45z\") pod \"payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:29.810357 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.810256 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:29.826277 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.826244 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7"] Nov 03 07:37:29.826763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.826749 4925 cpu_manager.go:414] "containerMap: removing container" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" Nov 03 07:37:29.826763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.826763 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" Nov 03 07:37:29.826893 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.826878 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9e5265-25ca-4578-9533-096c5818509d" containerName="istio-proxy" Nov 03 07:37:29.833517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.833487 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:29.883099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.882834 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7"] Nov 03 07:37:29.911680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.911622 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mnb\" (UniqueName: \"kubernetes.io/projected/be964d3c-36c1-43fd-b8da-b5483f6819ac-kube-api-access-r4mnb\") pod \"payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:29.911879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.911692 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be964d3c-36c1-43fd-b8da-b5483f6819ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:29.916051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.911749 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl45z\" (UniqueName: \"kubernetes.io/projected/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-kube-api-access-vl45z\") pod \"payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:29.916051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.912341 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:29.916051 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.913319 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:29.976433 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:29.976403 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl45z\" (UniqueName: \"kubernetes.io/projected/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-kube-api-access-vl45z\") pod \"payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:30.013643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.013613 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mnb\" (UniqueName: \"kubernetes.io/projected/be964d3c-36c1-43fd-b8da-b5483f6819ac-kube-api-access-r4mnb\") pod \"payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:30.013815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.013661 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be964d3c-36c1-43fd-b8da-b5483f6819ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:30.014416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.014394 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be964d3c-36c1-43fd-b8da-b5483f6819ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:30.017372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.017350 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:30.084783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.084706 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mnb\" (UniqueName: \"kubernetes.io/projected/be964d3c-36c1-43fd-b8da-b5483f6819ac-kube-api-access-r4mnb\") pod \"payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:30.155997 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.155920 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:30.190909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.190871 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43682d6b-e6ca-4ea0-be2f-91dc22fe735c" path="/var/lib/kubelet/pods/43682d6b-e6ca-4ea0-be2f-91dc22fe735c/volumes" Nov 03 07:37:30.191435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.191418 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b08ccb5-c6f4-4f73-b65a-aa547191ebb0" path="/var/lib/kubelet/pods/6b08ccb5-c6f4-4f73-b65a-aa547191ebb0/volumes" Nov 03 07:37:30.191834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.191820 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5e306e-987f-415d-994a-3d80a1673dff" path="/var/lib/kubelet/pods/7e5e306e-987f-415d-994a-3d80a1673dff/volumes" Nov 03 07:37:30.192602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.192583 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9e5265-25ca-4578-9533-096c5818509d" path="/var/lib/kubelet/pods/da9e5265-25ca-4578-9533-096c5818509d/volumes" Nov 03 07:37:30.397061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.397031 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54"] Nov 03 07:37:30.397487 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:30.397064 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ae1ea2_c7c3_4427_af01_f73d09e48cd2.slice/crio-66eea9de1d49b3768d455692ccfb4551ae49ffa7ba795d36f485b25b5b674a98 WatchSource:0}: Error finding container 66eea9de1d49b3768d455692ccfb4551ae49ffa7ba795d36f485b25b5b674a98: Status 404 returned error can't find the container with id 66eea9de1d49b3768d455692ccfb4551ae49ffa7ba795d36f485b25b5b674a98 Nov 03 07:37:30.398940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.398926 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:37:30.539059 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.539034 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7"] Nov 03 07:37:30.539851 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:30.539823 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe964d3c_36c1_43fd_b8da_b5483f6819ac.slice/crio-f0ba5d9d9816a40ea8265186b7230367e923d30a2624239d955ccf5d2b3950ff WatchSource:0}: Error finding container f0ba5d9d9816a40ea8265186b7230367e923d30a2624239d955ccf5d2b3950ff: Status 404 returned error can't find the container with id f0ba5d9d9816a40ea8265186b7230367e923d30a2624239d955ccf5d2b3950ff Nov 03 07:37:30.903878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:30.903841 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:31.151701 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:31.151663 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:31.195123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:31.195031 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" event={"ID":"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2","Type":"ContainerStarted","Data":"66eea9de1d49b3768d455692ccfb4551ae49ffa7ba795d36f485b25b5b674a98"} Nov 03 07:37:31.196753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:31.196718 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" event={"ID":"be964d3c-36c1-43fd-b8da-b5483f6819ac","Type":"ContainerStarted","Data":"f0ba5d9d9816a40ea8265186b7230367e923d30a2624239d955ccf5d2b3950ff"} Nov 03 07:37:31.953251 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:31.953213 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:32.144033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.143966 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:32.501868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.501833 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g"] Nov 03 07:37:32.517293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.517259 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:32.544239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.543280 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g"] Nov 03 07:37:32.589061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.586664 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50ae96-dockercfg-7mnpm\"" Nov 03 07:37:32.589061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.586714 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50ae96\"" Nov 03 07:37:32.659902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.659840 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm48x\" (UniqueName: \"kubernetes.io/projected/e4e875e6-0b31-447b-9069-2dd736fda882-kube-api-access-fm48x\") pod \"payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:32.660114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.659965 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e875e6-0b31-447b-9069-2dd736fda882-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:32.764215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.760988 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm48x\" (UniqueName: \"kubernetes.io/projected/e4e875e6-0b31-447b-9069-2dd736fda882-kube-api-access-fm48x\") pod \"payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:32.764215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.761089 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e875e6-0b31-447b-9069-2dd736fda882-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:32.764215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.761844 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e875e6-0b31-447b-9069-2dd736fda882-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:32.826625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.826591 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm48x\" (UniqueName: \"kubernetes.io/projected/e4e875e6-0b31-447b-9069-2dd736fda882-kube-api-access-fm48x\") pod \"payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:32.842417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:32.842389 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:33.246117 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:33.246080 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e875e6_0b31_447b_9069_2dd736fda882.slice/crio-cc768f10d0d862fddd82b5ba0b1b7a06129953c20e75f77a2b2b61af5b1ba306 WatchSource:0}: Error finding container cc768f10d0d862fddd82b5ba0b1b7a06129953c20e75f77a2b2b61af5b1ba306: Status 404 returned error can't find the container with id cc768f10d0d862fddd82b5ba0b1b7a06129953c20e75f77a2b2b61af5b1ba306 Nov 03 07:37:33.246562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:33.246484 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g"] Nov 03 07:37:34.233467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.233368 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" event={"ID":"be964d3c-36c1-43fd-b8da-b5483f6819ac","Type":"ContainerStarted","Data":"f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604"} Nov 03 07:37:34.233624 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.233551 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:34.235404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.235374 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" event={"ID":"e4e875e6-0b31-447b-9069-2dd736fda882","Type":"ContainerStarted","Data":"9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff"} Nov 03 07:37:34.235543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.235424 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" event={"ID":"e4e875e6-0b31-447b-9069-2dd736fda882","Type":"ContainerStarted","Data":"cc768f10d0d862fddd82b5ba0b1b7a06129953c20e75f77a2b2b61af5b1ba306"} Nov 03 07:37:34.235543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.235454 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:34.237181 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.237156 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" event={"ID":"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2","Type":"ContainerStarted","Data":"23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87"} Nov 03 07:37:34.237326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.237307 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:34.379539 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.379485 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" podStartSLOduration=1.97084451 podStartE2EDuration="5.379467931s" totalImagesPullingTime="3.408623421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:34.378250839 +0000 UTC m=+2314.311779501" watchObservedRunningTime="2026-11-03 07:37:34.379467931 +0000 UTC m=+2314.312996588" Nov 03 07:37:34.500279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.500174 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" podStartSLOduration=1.684135646 podStartE2EDuration="2.500160632s" totalImagesPullingTime="816.024986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:34.497730968 +0000 UTC m=+2314.431259625" watchObservedRunningTime="2026-11-03 07:37:34.500160632 +0000 UTC m=+2314.433689304" Nov 03 07:37:34.617712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:34.617663 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" podStartSLOduration=2.098170486 podStartE2EDuration="5.617648512s" totalImagesPullingTime="3.519478026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:34.617465729 +0000 UTC m=+2314.550994385" watchObservedRunningTime="2026-11-03 07:37:34.617648512 +0000 UTC m=+2314.551177168" Nov 03 07:37:36.235284 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:36.235245 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:36.474344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:36.474310 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:36.970158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:36.970121 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv"] Nov 03 07:37:36.975879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:36.975854 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.018983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.018955 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl"] Nov 03 07:37:37.022781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.022767 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.058420 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.058395 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-295fba7c-openshift-default-dockercfg-6kjds\"" Nov 03 07:37:37.070148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.070126 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv"] Nov 03 07:37:37.100064 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.100038 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl"] Nov 03 07:37:37.113101 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113073 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss262\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-kube-api-access-ss262\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.113258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113109 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.113258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113130 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbltq\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-kube-api-access-wbltq\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113217 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.113444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113321 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113383 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113420 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.113598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113476 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113508 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80af35f-cccf-496a-a7cb-177e3bc1f399-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113528 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.113598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113556 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113694 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113749 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6452b9bc-2048-4801-b00a-9e680d27ed5b-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.113798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113777 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113848 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.113954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113880 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.113954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113913 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.114172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.113984 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215223 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215191 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215231 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215251 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80af35f-cccf-496a-a7cb-177e3bc1f399-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215370 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215411 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215442 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215552 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6452b9bc-2048-4801-b00a-9e680d27ed5b-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215595 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215630 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215673 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215706 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215713 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215754 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215777 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215825 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215891 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss262\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-kube-api-access-ss262\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215930 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215960 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbltq\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-kube-api-access-wbltq\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.215985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215969 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.215985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.215978 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80af35f-cccf-496a-a7cb-177e3bc1f399-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.216358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216072 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.216358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216104 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.216358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216106 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.216358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216142 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.216358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216252 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6452b9bc-2048-4801-b00a-9e680d27ed5b-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.216570 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216398 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.216570 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216427 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.216570 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.216496 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.217889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.217856 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.218521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.218502 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.219078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.219056 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.219166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.219078 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.280513 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.280448 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss262\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-kube-api-access-ss262\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.280934 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.280868 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.281058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.281031 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.281437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.281408 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbltq\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-kube-api-access-wbltq\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.292667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.292461 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:37.335952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.335915 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:37.728794 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:37.728753 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80af35f_cccf_496a_a7cb_177e3bc1f399.slice/crio-b4e7b6a740b65a8a953f58b9ba647bc1a4088f8696e48d5f4d4907090fa5bae1 WatchSource:0}: Error finding container b4e7b6a740b65a8a953f58b9ba647bc1a4088f8696e48d5f4d4907090fa5bae1: Status 404 returned error can't find the container with id b4e7b6a740b65a8a953f58b9ba647bc1a4088f8696e48d5f4d4907090fa5bae1 Nov 03 07:37:37.728927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.728907 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv"] Nov 03 07:37:37.731439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.731404 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:37.731527 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.731461 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:37.731527 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.731488 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:37.783501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.783461 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl"] Nov 03 07:37:37.794184 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:37.794159 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6452b9bc_2048_4801_b00a_9e680d27ed5b.slice/crio-55918ca0330b0a586556ef0019748a20506429d077910bae696d9acdc706bb54 WatchSource:0}: Error finding container 55918ca0330b0a586556ef0019748a20506429d077910bae696d9acdc706bb54: Status 404 returned error can't find the container with id 55918ca0330b0a586556ef0019748a20506429d077910bae696d9acdc706bb54 Nov 03 07:37:37.797028 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.796979 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:37.797103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.797065 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:37.797103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:37.797096 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:38.269735 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.269694 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" event={"ID":"6452b9bc-2048-4801-b00a-9e680d27ed5b","Type":"ContainerStarted","Data":"a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b"} Nov 03 07:37:38.269902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.269767 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" event={"ID":"6452b9bc-2048-4801-b00a-9e680d27ed5b","Type":"ContainerStarted","Data":"55918ca0330b0a586556ef0019748a20506429d077910bae696d9acdc706bb54"} Nov 03 07:37:38.271885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.271851 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" event={"ID":"a80af35f-cccf-496a-a7cb-177e3bc1f399","Type":"ContainerStarted","Data":"714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57"} Nov 03 07:37:38.272055 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.271909 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" event={"ID":"a80af35f-cccf-496a-a7cb-177e3bc1f399","Type":"ContainerStarted","Data":"b4e7b6a740b65a8a953f58b9ba647bc1a4088f8696e48d5f4d4907090fa5bae1"} Nov 03 07:37:38.293711 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.293683 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:38.294938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.294909 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 07:37:38.295076 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.294966 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 07:37:38.336062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.336042 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:38.337194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.337172 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 07:37:38.337284 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.337219 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 07:37:38.395339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.395280 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podStartSLOduration=2.3952600459999998 podStartE2EDuration="2.395260046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:38.393762305 +0000 UTC m=+2318.327290961" watchObservedRunningTime="2026-11-03 07:37:38.395260046 +0000 UTC m=+2318.328788706" Nov 03 07:37:38.520574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:38.520471 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podStartSLOduration=2.520459099 podStartE2EDuration="2.520459099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:38.517734872 +0000 UTC m=+2318.451263528" watchObservedRunningTime="2026-11-03 07:37:38.520459099 +0000 UTC m=+2318.453987755" Nov 03 07:37:39.178304 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:39.178235 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:37:39.298942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:39.298909 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 07:37:39.299553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:39.299524 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 07:37:39.323741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:39.323646 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:39.339850 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:39.339189 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 07:37:39.339850 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:39.339286 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 07:37:39.536094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:39.530803 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:40.299036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:40.298433 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 07:37:40.299036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:40.298500 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 07:37:40.336414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:40.336352 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 07:37:40.336587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:40.336435 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 07:37:41.305055 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:41.298103 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 07:37:41.305055 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:41.298164 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 07:37:41.336967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:41.336936 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 07:37:41.337163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:41.336989 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 07:37:41.605215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:41.605090 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:42.297398 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:42.297352 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 07:37:42.297586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:42.297421 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 07:37:42.336227 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:42.336199 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 07:37:42.336672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:42.336648 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 07:37:43.298037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:43.297402 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 07:37:43.298037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:43.297479 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 07:37:43.336544 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:43.336512 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 07:37:43.336940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:43.336578 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 07:37:44.011037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:44.010762 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:44.293276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:44.293185 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 07:37:44.293276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:44.293247 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 07:37:44.336494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:44.336458 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 07:37:44.336658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:44.336521 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 07:37:45.249116 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.249086 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:37:45.250717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.250693 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:37:45.252123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.252103 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:37:45.298630 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.298600 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:45.298928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.298898 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:45.299965 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.299945 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:37:45.340802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.340779 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:45.341173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.341050 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:45.341870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:45.341852 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:37:46.426002 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.425969 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-32f0e5"] Nov 03 07:37:46.430990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.430963 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:46.472574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.472528 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqj2\" (UniqueName: \"kubernetes.io/projected/b09958c1-f449-4cd1-826c-1b90887c6573-kube-api-access-gqqj2\") pod \"test-curl-7135-32f0e5\" (UID: \"b09958c1-f449-4cd1-826c-1b90887c6573\") " pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:46.485806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.485774 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-32f0e5"] Nov 03 07:37:46.486921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.486886 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mnr67\"" Nov 03 07:37:46.574779 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.574728 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqj2\" (UniqueName: \"kubernetes.io/projected/b09958c1-f449-4cd1-826c-1b90887c6573-kube-api-access-gqqj2\") pod \"test-curl-7135-32f0e5\" (UID: \"b09958c1-f449-4cd1-826c-1b90887c6573\") " pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:46.640515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.640486 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqj2\" (UniqueName: \"kubernetes.io/projected/b09958c1-f449-4cd1-826c-1b90887c6573-kube-api-access-gqqj2\") pod \"test-curl-7135-32f0e5\" (UID: \"b09958c1-f449-4cd1-826c-1b90887c6573\") " pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:46.745833 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:46.745743 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:47.052345 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:47.052243 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:47.129794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:47.129764 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-32f0e5"] Nov 03 07:37:47.131156 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:47.131129 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb09958c1_f449_4cd1_826c_1b90887c6573.slice/crio-532dba18333d345ce9c77b2ff349cf8245a7e4e71d4cbf9d60ccd2684437fd7a WatchSource:0}: Error finding container 532dba18333d345ce9c77b2ff349cf8245a7e4e71d4cbf9d60ccd2684437fd7a: Status 404 returned error can't find the container with id 532dba18333d345ce9c77b2ff349cf8245a7e4e71d4cbf9d60ccd2684437fd7a Nov 03 07:37:47.426553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:47.426507 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-32f0e5" event={"ID":"b09958c1-f449-4cd1-826c-1b90887c6573","Type":"ContainerStarted","Data":"102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d"} Nov 03 07:37:47.427069 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:47.426563 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-32f0e5" event={"ID":"b09958c1-f449-4cd1-826c-1b90887c6573","Type":"ContainerStarted","Data":"532dba18333d345ce9c77b2ff349cf8245a7e4e71d4cbf9d60ccd2684437fd7a"} Nov 03 07:37:47.544686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:47.544638 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-32f0e5" podStartSLOduration=1.544624208 podStartE2EDuration="1.544624208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:47.543456763 +0000 UTC m=+2327.476985418" watchObservedRunningTime="2026-11-03 07:37:47.544624208 +0000 UTC m=+2327.478152863" Nov 03 07:37:48.281856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:48.281819 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:37:49.088004 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:49.087973 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-32f0e5"] Nov 03 07:37:49.097332 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:49.097303 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-32f0e5"] Nov 03 07:37:49.443071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:49.442949 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-32f0e5" podUID="b09958c1-f449-4cd1-826c-1b90887c6573" containerName="test-curl-7135-32f0e5" containerID="cri-o://102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d" gracePeriod=2 Nov 03 07:37:49.495346 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:49.495300 4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-32f0e5\" is forbidden: User \"system:node:ip-10-0-136-184.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-184.us-west-1.compute.internal' and this object" podUID="b09958c1-f449-4cd1-826c-1b90887c6573" pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:50.240116 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:50.240079 4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-32f0e5\" is forbidden: User \"system:node:ip-10-0-136-184.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-184.us-west-1.compute.internal' and this object" podUID="b09958c1-f449-4cd1-826c-1b90887c6573" pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:51.700553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:51.700493 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:51.842746 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:51.842717 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09958c1-f449-4cd1-826c-1b90887c6573-kube-api-access-gqqj2\" (UniqueName: \"kubernetes.io/projected/b09958c1-f449-4cd1-826c-1b90887c6573-kube-api-access-gqqj2\") pod \"b09958c1-f449-4cd1-826c-1b90887c6573\" (UID: \"b09958c1-f449-4cd1-826c-1b90887c6573\") " Nov 03 07:37:51.844949 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:51.844920 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09958c1-f449-4cd1-826c-1b90887c6573-kube-api-access-gqqj2" pod "b09958c1-f449-4cd1-826c-1b90887c6573" (UID: "b09958c1-f449-4cd1-826c-1b90887c6573"). InnerVolumeSpecName "kube-api-access-gqqj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:51.944547 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:51.944518 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqj2\" (UniqueName: \"kubernetes.io/projected/b09958c1-f449-4cd1-826c-1b90887c6573-kube-api-access-gqqj2\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:52.109451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.109422 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml"] Nov 03 07:37:52.109986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.109974 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b09958c1-f449-4cd1-826c-1b90887c6573" containerName="test-curl-7135-32f0e5" Nov 03 07:37:52.110053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.109988 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09958c1-f449-4cd1-826c-1b90887c6573" containerName="test-curl-7135-32f0e5" Nov 03 07:37:52.110134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.110124 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09958c1-f449-4cd1-826c-1b90887c6573" containerName="test-curl-7135-32f0e5" Nov 03 07:37:52.113470 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.113454 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.191947 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.191913 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09958c1-f449-4cd1-826c-1b90887c6573" path="/var/lib/kubelet/pods/b09958c1-f449-4cd1-826c-1b90887c6573/volumes" Nov 03 07:37:52.248588 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.248554 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.248775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.248614 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.248775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.248738 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.248916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.248852 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.248972 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.248934 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.249050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.248981 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.249118 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.249035 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8cn\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-kube-api-access-5x8cn\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.249315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.249277 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.249441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.249327 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.262340 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.262311 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:52.329271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.329242 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml"] Nov 03 07:37:52.350854 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.350823 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.350863 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8cn\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-kube-api-access-5x8cn\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.350900 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.350930 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351241 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351069 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351241 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351147 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351241 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351225 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351292 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351355 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351376 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351422 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351510 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.351725 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.351702 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.352386 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.352357 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.353880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.353852 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.353981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.353913 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.415396 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.415324 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.415533 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.415464 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8cn\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-kube-api-access-5x8cn\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.427623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.427599 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:52.469099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.469058 4925 generic.go:356] "Generic (PLEG): container finished" podID="b09958c1-f449-4cd1-826c-1b90887c6573" containerID="102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d" exitCode=137 Nov 03 07:37:52.469313 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.469151 4925 scope.go:122] "RemoveContainer" containerID="102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d" Nov 03 07:37:52.469425 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.469176 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-32f0e5" Nov 03 07:37:52.479929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.479906 4925 scope.go:122] "RemoveContainer" containerID="102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d" Nov 03 07:37:52.480274 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:37:52.480247 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d\": container with ID starting with 102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d not found: ID does not exist" containerID="102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d" Nov 03 07:37:52.480371 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.480287 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d"} err="failed to get container status \"102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d\": rpc error: code = NotFound desc = could not find container \"102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d\": container with ID starting with 102ddd2ef3d483978519ea31aaaaa3452714dd57e2fb513577e88d66fdfc460d not found: ID does not exist" Nov 03 07:37:52.872734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.871693 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml"] Nov 03 07:37:52.873902 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:37:52.873870 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc42f4e_2481_402d_84c1_7a7e3f81f105.slice/crio-e4654db6a0c14f3d338831ba09976a7d04d51d8729ba79f41d6959464ffd53c5 WatchSource:0}: Error finding container e4654db6a0c14f3d338831ba09976a7d04d51d8729ba79f41d6959464ffd53c5: Status 404 returned error can't find the container with id e4654db6a0c14f3d338831ba09976a7d04d51d8729ba79f41d6959464ffd53c5 Nov 03 07:37:52.879097 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.879062 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:52.879205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.879135 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:52.879205 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:52.879164 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:37:53.491158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:53.491123 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" event={"ID":"2fc42f4e-2481-402d-84c1-7a7e3f81f105","Type":"ContainerStarted","Data":"cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570"} Nov 03 07:37:53.491339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:53.491175 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" event={"ID":"2fc42f4e-2481-402d-84c1-7a7e3f81f105","Type":"ContainerStarted","Data":"e4654db6a0c14f3d338831ba09976a7d04d51d8729ba79f41d6959464ffd53c5"} Nov 03 07:37:53.621658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:53.621597 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" podStartSLOduration=1.621583388 podStartE2EDuration="1.621583388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:53.620766685 +0000 UTC m=+2333.554295342" watchObservedRunningTime="2026-11-03 07:37:53.621583388 +0000 UTC m=+2333.555112043" Nov 03 07:37:54.427798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:54.427768 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:54.429515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:54.429487 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:37:54.429657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:54.429536 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 07:37:55.428422 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:55.428381 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:37:55.428875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:55.428448 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 07:37:56.427950 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:56.427915 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:37:56.428143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:56.427969 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 07:37:57.102255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:57.102163 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:57.429755 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:57.429719 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:37:57.430049 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:57.429996 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 07:37:58.427980 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:58.427940 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:37:58.428487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:58.428038 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 07:37:58.734734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:58.734631 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:59.097991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:59.097951 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:59.352103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:59.351983 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:37:59.432636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:59.432611 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:59.433084 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:59.432988 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:59.433671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:59.433654 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:37:59.753691 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:37:59.753540 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:00.993679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:00.993638 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:01.184570 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:01.184535 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:04.048730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:04.048676 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:04.477164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:04.477071 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:06.769683 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:06.769645 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:07.023767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.023629 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:07.254841 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.254784 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth"] Nov 03 07:38:07.269803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.269760 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.299837 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.299802 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wskl\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-kube-api-access-8wskl\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.299864 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.299977 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.300074 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.300135 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.300162 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.300229 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.300255 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c9ddd3-3089-48ca-b64e-b429f63b472b-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.300373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.300283 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.304142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.304115 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:07.332323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.332296 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth"] Nov 03 07:38:07.401838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.401800 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.401838 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.401844 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.401893 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.401915 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c9ddd3-3089-48ca-b64e-b429f63b472b-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.401930 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.401981 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wskl\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-kube-api-access-8wskl\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402005 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402078 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402133 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402510 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402385 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402510 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402415 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402510 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402481 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402492 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.402985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.402922 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c9ddd3-3089-48ca-b64e-b429f63b472b-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.404333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.404312 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.404791 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.404774 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.468571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.468540 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.468778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.468755 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wskl\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-kube-api-access-8wskl\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.588242 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.588209 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:07.630201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:07.630161 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:08.024987 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:38:08.024954 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c9ddd3_3089_48ca_b64e_b429f63b472b.slice/crio-ce61e682cedf842285ef4b45c286d1648c95d71ac92971c2f6e7730aa4966e40 WatchSource:0}: Error finding container ce61e682cedf842285ef4b45c286d1648c95d71ac92971c2f6e7730aa4966e40: Status 404 returned error can't find the container with id ce61e682cedf842285ef4b45c286d1648c95d71ac92971c2f6e7730aa4966e40 Nov 03 07:38:08.025364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:08.025199 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth"] Nov 03 07:38:08.027515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:08.027484 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:38:08.027602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:08.027547 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:38:08.027602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:08.027580 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:38:08.608157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:08.608112 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" event={"ID":"76c9ddd3-3089-48ca-b64e-b429f63b472b","Type":"ContainerStarted","Data":"0bf9af95ec884d6871f22fe2f6cc9981acf2fd33673a81b7cfeb09f4e91fe67e"} Nov 03 07:38:08.608157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:08.608175 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" event={"ID":"76c9ddd3-3089-48ca-b64e-b429f63b472b","Type":"ContainerStarted","Data":"ce61e682cedf842285ef4b45c286d1648c95d71ac92971c2f6e7730aa4966e40"} Nov 03 07:38:08.782119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:08.782063 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" podStartSLOduration=1.782043615 podStartE2EDuration="1.782043615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:38:08.779562835 +0000 UTC m=+2348.713091483" watchObservedRunningTime="2026-11-03 07:38:08.782043615 +0000 UTC m=+2348.715572276" Nov 03 07:38:09.139485 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:38:09.139456 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:38:09.590027 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:09.589972 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:09.591605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:09.591573 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 07:38:09.591729 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:09.591632 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 07:38:10.002905 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:10.002780 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:10.588832 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:10.588794 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 07:38:10.589270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:10.588853 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 07:38:11.589053 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:11.588988 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 07:38:11.589455 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:11.589094 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 07:38:12.588896 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:12.588859 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 07:38:12.589114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:12.588926 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 07:38:12.769038 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:12.768561 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:12.989859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:12.989767 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:13.182551 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:13.182519 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:13.588638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:13.588598 4925 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 07:38:13.588862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:13.588667 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 07:38:14.644432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:14.644401 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:14.644921 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:14.644875 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:14.645700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:14.645677 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:38:15.164148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.164116 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96"] Nov 03 07:38:15.178229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.178196 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.228311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.228274 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96"] Nov 03 07:38:15.235278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.235238 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a8983d-serving-cert\"" Nov 03 07:38:15.340596 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.340559 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.340793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.340611 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9wqr\" (UniqueName: \"kubernetes.io/projected/972f9a40-0986-497e-8720-5876a4dbfd5c-kube-api-access-k9wqr\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.340842 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.340807 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f9a40-0986-497e-8720-5876a4dbfd5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.340897 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.340882 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-metrics-certs\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.442339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.442261 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-metrics-certs\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.442491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.442362 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.442491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.442396 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9wqr\" (UniqueName: \"kubernetes.io/projected/972f9a40-0986-497e-8720-5876a4dbfd5c-kube-api-access-k9wqr\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.442491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.442475 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f9a40-0986-497e-8720-5876a4dbfd5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.442606 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:38:15.442514 4925 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-serving-cert: secret "maas-api-e2e-api-a-a8983d-serving-cert" not found Nov 03 07:38:15.442606 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:38:15.442581 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls podName:972f9a40-0986-497e-8720-5876a4dbfd5c nodeName:}" failed. No retries permitted until 2026-11-03 07:38:15.942562114 +0000 UTC m=+2355.876090752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls") pod "maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" (UID: "972f9a40-0986-497e-8720-5876a4dbfd5c") : secret "maas-api-e2e-api-a-a8983d-serving-cert" not found Nov 03 07:38:15.443264 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.443238 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f9a40-0986-497e-8720-5876a4dbfd5c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.445126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.445099 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-metrics-certs\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.485478 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.485440 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:15.513983 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.513950 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9wqr\" (UniqueName: \"kubernetes.io/projected/972f9a40-0986-497e-8720-5876a4dbfd5c-kube-api-access-k9wqr\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.724104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.723991 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:15.911638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.911608 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:15.949096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.949064 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:15.951766 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:15.951740 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls\") pod \"maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:16.096106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:16.096056 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:16.480342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:16.480310 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96"] Nov 03 07:38:16.481491 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:38:16.480961 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972f9a40_0986_497e_8720_5876a4dbfd5c.slice/crio-9dd24310804becf5c392f92f4cc3914bb0dbe344e7e58fe2bb94fdd5c4295d5d WatchSource:0}: Error finding container 9dd24310804becf5c392f92f4cc3914bb0dbe344e7e58fe2bb94fdd5c4295d5d: Status 404 returned error can't find the container with id 9dd24310804becf5c392f92f4cc3914bb0dbe344e7e58fe2bb94fdd5c4295d5d Nov 03 07:38:16.665945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:16.665910 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" event={"ID":"972f9a40-0986-497e-8720-5876a4dbfd5c","Type":"ContainerStarted","Data":"9dd24310804becf5c392f92f4cc3914bb0dbe344e7e58fe2bb94fdd5c4295d5d"} Nov 03 07:38:17.676458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:17.676354 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" event={"ID":"972f9a40-0986-497e-8720-5876a4dbfd5c","Type":"ContainerStarted","Data":"0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5"} Nov 03 07:38:17.676458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:17.676428 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:17.801732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:17.801671 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" podStartSLOduration=1.943483071 podStartE2EDuration="2.80165254s" totalImagesPullingTime="858.169469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:38:17.797550699 +0000 UTC m=+2357.731079367" watchObservedRunningTime="2026-11-03 07:38:17.80165254 +0000 UTC m=+2357.735181197" Nov 03 07:38:18.578695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:18.578660 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:18.812616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:18.812564 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:20.729738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:20.729701 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:23.690856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:23.690826 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:38:23.897558 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:23.897507 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:24.136117 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:24.136085 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:25.669786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:25.669749 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:25.875421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:25.875383 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:30.680481 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:30.680444 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:30.917975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:30.917924 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:31.109680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:31.109641 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:31.298358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:31.298327 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz"] Nov 03 07:38:31.298601 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:31.298573 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" containerID="cri-o://2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65" gracePeriod=30 Nov 03 07:38:35.428287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:35.428248 4925 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:38:35.428712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:35.428322 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:38:35.840664 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:35.840612 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:36.587487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.587458 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:38:36.727089 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727057 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-certs\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727099 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-kube-api-access-8rcr2\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-kube-api-access-8rcr2\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727133 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-envoy\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727284 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-socket\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727316 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-podinfo\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727378 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-token\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-token\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727453 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-data\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727460 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-certs" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.727639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727503 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istiod-ca-cert\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.727639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727593 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-credential-socket\") pod \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\" (UID: \"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd\") " Nov 03 07:38:36.728198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.727895 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-data" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.728198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728091 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istiod-ca-cert" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:38:36.728198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728179 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-credential-socket" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.728549 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728394 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-socket" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.728927 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728904 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.729073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728930 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.729073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728944 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.729073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728956 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.729073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.728968 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.730315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.730288 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-podinfo" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:38:36.730426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.730326 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-envoy" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.730426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.730349 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-kube-api-access-8rcr2" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "kube-api-access-8rcr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:38:36.730673 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.730633 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-token" pod "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" (UID: "59f18c73-b9a7-4250-bb6e-2f88df5cb4bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:38:36.812413 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.812364 4925 generic.go:356] "Generic (PLEG): container finished" podID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerID="2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65" exitCode=0 Nov 03 07:38:36.812598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.812420 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" Nov 03 07:38:36.812598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.812437 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" event={"ID":"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd","Type":"ContainerDied","Data":"2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65"} Nov 03 07:38:36.812598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.812510 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz" event={"ID":"59f18c73-b9a7-4250-bb6e-2f88df5cb4bd","Type":"ContainerDied","Data":"b67a566d7c84505427b8082c1d41f6863ecba99a8b5bbe386adb7da26005a228"} Nov 03 07:38:36.812598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.812542 4925 scope.go:122] "RemoveContainer" containerID="2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65" Nov 03 07:38:36.824698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.824678 4925 scope.go:122] "RemoveContainer" containerID="2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65" Nov 03 07:38:36.824996 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:38:36.824977 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65\": container with ID starting with 2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65 not found: ID does not exist" containerID="2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65" Nov 03 07:38:36.825175 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.825041 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65"} err="failed to get container status \"2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65\": rpc error: code = NotFound desc = could not find container \"2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65\": container with ID starting with 2a27662226d4ff225f5ecb5f21c476ab2345fc5cb49d7f65363aca08e7dd8c65 not found: ID does not exist" Nov 03 07:38:36.830442 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.830416 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.830545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.830446 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rcr2\" (UniqueName: \"kubernetes.io/projected/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-kube-api-access-8rcr2\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.830545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.830461 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.830545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.830475 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.986353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.986306 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz"] Nov 03 07:38:36.992304 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:36.992277 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz"] Nov 03 07:38:38.192214 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:38.192183 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" path="/var/lib/kubelet/pods/59f18c73-b9a7-4250-bb6e-2f88df5cb4bd/volumes" Nov 03 07:38:39.139394 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:38:39.139361 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:38:40.524314 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.524243 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn"] Nov 03 07:38:40.524852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.524838 4925 cpu_manager.go:414] "containerMap: removing container" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" Nov 03 07:38:40.524898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.524853 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" Nov 03 07:38:40.524955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.524946 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f18c73-b9a7-4250-bb6e-2f88df5cb4bd" containerName="istio-proxy" Nov 03 07:38:40.529645 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.529625 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.635230 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.635206 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dcdb23-serving-cert\"" Nov 03 07:38:40.639502 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.639470 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn"] Nov 03 07:38:40.692365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.692334 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74065d0-b412-4763-8da5-9ec45c612370-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.692553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.692370 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-metrics-certs\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.692553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.692503 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/f74065d0-b412-4763-8da5-9ec45c612370-kube-api-access-x2kkr\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.692553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.692551 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-maas-api-tls\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.784793 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.784704 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:40.802056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.797864 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74065d0-b412-4763-8da5-9ec45c612370-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.802056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.797909 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-metrics-certs\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.802056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.797983 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/f74065d0-b412-4763-8da5-9ec45c612370-kube-api-access-x2kkr\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.802056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.798056 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-maas-api-tls\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.802056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.799342 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74065d0-b412-4763-8da5-9ec45c612370-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.803968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.802849 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-maas-api-tls\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.804399 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.804377 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-metrics-certs\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.862562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.862521 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/f74065d0-b412-4763-8da5-9ec45c612370-kube-api-access-x2kkr\") pod \"maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:40.970587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:40.970550 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:41.143437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:41.143410 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:41.536702 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:38:41.536673 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74065d0_b412_4763_8da5_9ec45c612370.slice/crio-f71b072b9f6e27ff6f9b49769234c102059d80ef90e20a1caa60a05086b06f79 WatchSource:0}: Error finding container f71b072b9f6e27ff6f9b49769234c102059d80ef90e20a1caa60a05086b06f79: Status 404 returned error can't find the container with id f71b072b9f6e27ff6f9b49769234c102059d80ef90e20a1caa60a05086b06f79 Nov 03 07:38:41.538523 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:41.538496 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn"] Nov 03 07:38:41.850806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:41.850761 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" event={"ID":"f74065d0-b412-4763-8da5-9ec45c612370","Type":"ContainerStarted","Data":"f71b072b9f6e27ff6f9b49769234c102059d80ef90e20a1caa60a05086b06f79"} Nov 03 07:38:42.860436 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:42.860393 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" event={"ID":"f74065d0-b412-4763-8da5-9ec45c612370","Type":"ContainerStarted","Data":"38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18"} Nov 03 07:38:42.860924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:42.860533 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:42.987062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:42.986976 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" podStartSLOduration=2.078950266 podStartE2EDuration="2.986955261s" totalImagesPullingTime="908.004995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:38:42.982169299 +0000 UTC m=+2382.915697967" watchObservedRunningTime="2026-11-03 07:38:42.986955261 +0000 UTC m=+2382.920483919" Nov 03 07:38:45.917828 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:45.917783 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:48.874126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:48.874097 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:38:50.160511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:50.160475 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:50.373247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:50.373219 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:51.079447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:51.079414 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:51.682879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:51.682842 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:51.875536 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:51.875500 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:52.070865 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:52.070830 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:52.271134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:52.271098 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:52.475593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:52.475501 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:52.675361 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:52.675327 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:53.035495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:53.035458 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:54.581119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:54.581085 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:38:57.650445 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:57.650402 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:58.069671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:58.069635 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:59.174586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:59.174552 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:38:59.410924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:38:59.410884 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:01.801135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:01.801107 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:39:01.808866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:01.808836 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:39:01.809330 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:01.809311 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:39:01.812569 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:01.812550 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:39:01.865395 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:01.865364 4925 scope.go:122] "RemoveContainer" containerID="fee7f056cde181231666e3320d4b605475b29ef3a75cfeb42be2ae759c4e532a" Nov 03 07:39:02.641063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:02.641025 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:04.405162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:04.405121 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:06.380071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:06.380023 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:08.073359 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:08.073321 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:09.139781 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:39:09.139746 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:39:11.325552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:11.325519 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm"] Nov 03 07:39:11.513817 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:11.513776 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:13.473982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:13.473936 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:18.909667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:18.904106 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:20.547367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:20.547325 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:23.931731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:23.931685 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:26.346987 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.346915 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" containerID="cri-o://383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984" gracePeriod=45 Nov 03 07:39:26.501745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.501697 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": dial tcp 10.132.0.58:8000: connect: connection refused" Nov 03 07:39:26.630426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.630395 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:39:26.763850 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.763809 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-dshm\") pod \"4a870ab4-1945-477a-b018-1fe6fe7f1047\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " Nov 03 07:39:26.764072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.763858 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-tmp-dir\") pod \"4a870ab4-1945-477a-b018-1fe6fe7f1047\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " Nov 03 07:39:26.764072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.763922 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a870ab4-1945-477a-b018-1fe6fe7f1047-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a870ab4-1945-477a-b018-1fe6fe7f1047-tls-certs\") pod \"4a870ab4-1945-477a-b018-1fe6fe7f1047\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " Nov 03 07:39:26.764072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.764046 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-home\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-home\") pod \"4a870ab4-1945-477a-b018-1fe6fe7f1047\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " Nov 03 07:39:26.764245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.764100 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a870ab4-1945-477a-b018-1fe6fe7f1047-kube-api-access-j4zvh\" (UniqueName: \"kubernetes.io/projected/4a870ab4-1945-477a-b018-1fe6fe7f1047-kube-api-access-j4zvh\") pod \"4a870ab4-1945-477a-b018-1fe6fe7f1047\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " Nov 03 07:39:26.764245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.764197 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-model-cache\") pod \"4a870ab4-1945-477a-b018-1fe6fe7f1047\" (UID: \"4a870ab4-1945-477a-b018-1fe6fe7f1047\") " Nov 03 07:39:26.767260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.766923 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-model-cache" pod "4a870ab4-1945-477a-b018-1fe6fe7f1047" (UID: "4a870ab4-1945-477a-b018-1fe6fe7f1047"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.770463 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.770156 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-home" pod "4a870ab4-1945-477a-b018-1fe6fe7f1047" (UID: "4a870ab4-1945-477a-b018-1fe6fe7f1047"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.770463 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.770176 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-tmp-dir" pod "4a870ab4-1945-477a-b018-1fe6fe7f1047" (UID: "4a870ab4-1945-477a-b018-1fe6fe7f1047"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.771281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.771245 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-dshm" pod "4a870ab4-1945-477a-b018-1fe6fe7f1047" (UID: "4a870ab4-1945-477a-b018-1fe6fe7f1047"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.772736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.772702 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a870ab4-1945-477a-b018-1fe6fe7f1047-kube-api-access-j4zvh" pod "4a870ab4-1945-477a-b018-1fe6fe7f1047" (UID: "4a870ab4-1945-477a-b018-1fe6fe7f1047"). InnerVolumeSpecName "kube-api-access-j4zvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:39:26.772933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.772903 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a870ab4-1945-477a-b018-1fe6fe7f1047-tls-certs" pod "4a870ab4-1945-477a-b018-1fe6fe7f1047" (UID: "4a870ab4-1945-477a-b018-1fe6fe7f1047"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:39:26.866152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.866068 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a870ab4-1945-477a-b018-1fe6fe7f1047-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.866152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.866095 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.866152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.866105 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zvh\" (UniqueName: \"kubernetes.io/projected/4a870ab4-1945-477a-b018-1fe6fe7f1047-kube-api-access-j4zvh\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.866152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.866114 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.866152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.866124 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.866152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:26.866151 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a870ab4-1945-477a-b018-1fe6fe7f1047-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.220489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.220415 4925 generic.go:356] "Generic (PLEG): container finished" podID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerID="383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984" exitCode=0 Nov 03 07:39:27.220489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.220462 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" Nov 03 07:39:27.220698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.220504 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" event={"ID":"4a870ab4-1945-477a-b018-1fe6fe7f1047","Type":"ContainerDied","Data":"383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984"} Nov 03 07:39:27.220698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.220548 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm" event={"ID":"4a870ab4-1945-477a-b018-1fe6fe7f1047","Type":"ContainerDied","Data":"6ecfb970d9211049429e47d81513323cec257127423474bff1e5a4fc81ab19dd"} Nov 03 07:39:27.220698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.220565 4925 scope.go:122] "RemoveContainer" containerID="383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984" Nov 03 07:39:27.235204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.235185 4925 scope.go:122] "RemoveContainer" containerID="383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984" Nov 03 07:39:27.235508 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:39:27.235489 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984\": container with ID starting with 383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984 not found: ID does not exist" containerID="383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984" Nov 03 07:39:27.235575 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.235517 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984"} err="failed to get container status \"383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984\": rpc error: code = NotFound desc = could not find container \"383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984\": container with ID starting with 383237ac1b75ea5d0a55e90c0942637c84b8eb493849c291de062aee51bb0984 not found: ID does not exist" Nov 03 07:39:27.392901 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.392865 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm"] Nov 03 07:39:27.444020 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.443967 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/simulator-w6-3b41b9-kserve-8c97667bd-fh4wm"] Nov 03 07:39:27.889980 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:27.889929 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:28.195504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:28.195427 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" path="/var/lib/kubelet/pods/4a870ab4-1945-477a-b018-1fe6fe7f1047/volumes" Nov 03 07:39:28.932690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:28.932656 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:29.185963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:29.185870 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:29.381389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:29.381352 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:29.863986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:29.863940 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:30.520749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:30.520703 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:30.925034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:30.924538 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:31.087525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:31.087487 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:39:34.989039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:34.988222 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:35.370733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:35.370636 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:35.571658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:35.571626 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:36.030135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:36.030101 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:36.871279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:36.871238 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:37.107058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:37.107004 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:37.531340 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:37.531299 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:37.742996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:37.742954 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:39.157311 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:39:39.157243 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:39:40.520862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:40.520817 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:40.713635 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:40.713599 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:42.148144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.148114 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:39:42.148587 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.148179 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 07:39:42.363158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.363125 4925 generic.go:356] "Generic (PLEG): container finished" podID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerID="0a9cae86b01ff5f8b9c989c760d7a0b9eb486aafe25d6a803538dc26c0f671ab" exitCode=0 Nov 03 07:39:42.363330 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.363200 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" event={"ID":"a52ae06b-ed08-47b0-be5f-2e522283b659","Type":"ContainerDied","Data":"0a9cae86b01ff5f8b9c989c760d7a0b9eb486aafe25d6a803538dc26c0f671ab"} Nov 03 07:39:42.363632 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.363618 4925 scope.go:122] "RemoveContainer" containerID="0a9cae86b01ff5f8b9c989c760d7a0b9eb486aafe25d6a803538dc26c0f671ab" Nov 03 07:39:42.365162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.365135 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:39:42.365232 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.365204 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:39:42.365271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.365238 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:39:42.677381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.677302 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:42.918196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:42.918159 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:43.327203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:43.327172 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:43.380941 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:43.380905 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" event={"ID":"a52ae06b-ed08-47b0-be5f-2e522283b659","Type":"ContainerStarted","Data":"d469f17fc022c50332d68eae35b3ead7322fb164d620f46a9cde7c7219f42185"} Nov 03 07:39:43.532100 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:43.532063 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:39:43.532272 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:43.532222 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:39:43.532316 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:43.532266 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 07:39:43.739398 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:43.739307 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:44.385057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:44.384884 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:44.532005 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:44.531961 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:39:44.532250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:44.532048 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 07:39:44.740225 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:44.740135 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:45.532030 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:45.531969 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:39:45.532445 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:45.532059 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 07:39:46.118865 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:46.118822 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:46.532163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:46.532078 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:39:46.532163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:46.532134 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 07:39:47.532060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:47.531995 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:39:47.532246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:47.532083 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 07:39:47.869666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:47.869631 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:48.075131 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:48.075074 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:48.532554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:48.532514 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:39:48.532980 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:48.532596 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 07:39:49.536567 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:49.536537 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:39:49.537056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:49.537038 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:39:49.537619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:49.537600 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:39:50.292405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:50.292360 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:50.503584 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:50.503544 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:51.758093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:51.758056 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:54.990401 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:54.990353 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:55.216860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:55.216820 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:55.463407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:55.463368 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:56.867623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:56.867571 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:57.261128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:57.261000 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:59.662238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:59.662169 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:39:59.859932 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:39:59.859891 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:01.939524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:01.939479 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:02.824378 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:02.824329 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:04.752557 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:04.752525 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:04.995063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:04.995028 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:05.200035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.198447 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl"] Nov 03 07:40:05.200035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.198692 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" containerID="cri-o://a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b" gracePeriod=30 Nov 03 07:40:05.225963 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.225929 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml"] Nov 03 07:40:05.226217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.226192 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" containerID="cri-o://cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570" gracePeriod=30 Nov 03 07:40:05.238744 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.238714 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth"] Nov 03 07:40:05.239142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.238927 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" containerID="cri-o://0bf9af95ec884d6871f22fe2f6cc9981acf2fd33673a81b7cfeb09f4e91fe67e" gracePeriod=30 Nov 03 07:40:05.248478 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.248449 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv"] Nov 03 07:40:05.248662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.248644 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" containerID="cri-o://714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57" gracePeriod=30 Nov 03 07:40:05.278215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:05.271403 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:07.954029 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:07.952051 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:08.849423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:08.849388 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:09.139079 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:40:09.138989 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:40:10.569248 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.569217 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:40:10.615374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.615340 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:40:10.619282 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.619262 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:40:10.622158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.622135 4925 generic.go:356] "Generic (PLEG): container finished" podID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerID="0bf9af95ec884d6871f22fe2f6cc9981acf2fd33673a81b7cfeb09f4e91fe67e" exitCode=0 Nov 03 07:40:10.622267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.622210 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" event={"ID":"76c9ddd3-3089-48ca-b64e-b429f63b472b","Type":"ContainerDied","Data":"0bf9af95ec884d6871f22fe2f6cc9981acf2fd33673a81b7cfeb09f4e91fe67e"} Nov 03 07:40:10.623456 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.623436 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:40:10.623672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.623652 4925 generic.go:356] "Generic (PLEG): container finished" podID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerID="cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570" exitCode=0 Nov 03 07:40:10.623722 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.623681 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" event={"ID":"2fc42f4e-2481-402d-84c1-7a7e3f81f105","Type":"ContainerDied","Data":"cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570"} Nov 03 07:40:10.623722 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.623695 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" Nov 03 07:40:10.623799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.623729 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml" event={"ID":"2fc42f4e-2481-402d-84c1-7a7e3f81f105","Type":"ContainerDied","Data":"e4654db6a0c14f3d338831ba09976a7d04d51d8729ba79f41d6959464ffd53c5"} Nov 03 07:40:10.623799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.623760 4925 scope.go:122] "RemoveContainer" containerID="cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570" Nov 03 07:40:10.624968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.624949 4925 generic.go:356] "Generic (PLEG): container finished" podID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerID="a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b" exitCode=0 Nov 03 07:40:10.625075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.624989 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" Nov 03 07:40:10.625075 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.625060 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" event={"ID":"6452b9bc-2048-4801-b00a-9e680d27ed5b","Type":"ContainerDied","Data":"a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b"} Nov 03 07:40:10.625194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.625106 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl" event={"ID":"6452b9bc-2048-4801-b00a-9e680d27ed5b","Type":"ContainerDied","Data":"55918ca0330b0a586556ef0019748a20506429d077910bae696d9acdc706bb54"} Nov 03 07:40:10.626409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.626391 4925 generic.go:356] "Generic (PLEG): container finished" podID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerID="714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57" exitCode=0 Nov 03 07:40:10.626487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.626423 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" Nov 03 07:40:10.626487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.626433 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" event={"ID":"a80af35f-cccf-496a-a7cb-177e3bc1f399","Type":"ContainerDied","Data":"714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57"} Nov 03 07:40:10.626487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.626475 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv" event={"ID":"a80af35f-cccf-496a-a7cb-177e3bc1f399","Type":"ContainerDied","Data":"b4e7b6a740b65a8a953f58b9ba647bc1a4088f8696e48d5f4d4907090fa5bae1"} Nov 03 07:40:10.633646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.633626 4925 scope.go:122] "RemoveContainer" containerID="cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570" Nov 03 07:40:10.633936 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:40:10.633918 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570\": container with ID starting with cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570 not found: ID does not exist" containerID="cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570" Nov 03 07:40:10.634054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.633944 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570"} err="failed to get container status \"cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570\": rpc error: code = NotFound desc = could not find container \"cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570\": container with ID starting with cbfa24d2e6cf685ea37b79a8b6875904dabc685f9963421cfdffaa4437d76570 not found: ID does not exist" Nov 03 07:40:10.634054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.633958 4925 scope.go:122] "RemoveContainer" containerID="a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b" Nov 03 07:40:10.641221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641201 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-socket\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641325 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641255 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-credential-socket\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641325 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641319 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-certs\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641342 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6452b9bc-2048-4801-b00a-9e680d27ed5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6452b9bc-2048-4801-b00a-9e680d27ed5b-istiod-ca-cert\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641392 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-data\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641477 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-podinfo\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641487 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-socket" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.641554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641541 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-token\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-token\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641704 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641576 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-credential-socket" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.641704 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641589 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-certs" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.641704 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641625 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-kube-api-access-ss262\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-kube-api-access-ss262\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641704 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641685 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-envoy\") pod \"6452b9bc-2048-4801-b00a-9e680d27ed5b\" (UID: \"6452b9bc-2048-4801-b00a-9e680d27ed5b\") " Nov 03 07:40:10.641985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.641795 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6452b9bc-2048-4801-b00a-9e680d27ed5b-istiod-ca-cert" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:10.642122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.642102 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-data" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.642269 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.642248 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.642327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.642275 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.642327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.642290 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.642327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.642304 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6452b9bc-2048-4801-b00a-9e680d27ed5b-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.642327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.642317 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.644464 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.644436 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-podinfo" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:10.644545 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.644525 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-token" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.644625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.644608 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-envoy" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.644872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.644855 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-kube-api-access-ss262" pod "6452b9bc-2048-4801-b00a-9e680d27ed5b" (UID: "6452b9bc-2048-4801-b00a-9e680d27ed5b"). InnerVolumeSpecName "kube-api-access-ss262". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.645690 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.645675 4925 scope.go:122] "RemoveContainer" containerID="a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b" Nov 03 07:40:10.645968 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:40:10.645952 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b\": container with ID starting with a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b not found: ID does not exist" containerID="a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b" Nov 03 07:40:10.646039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.645975 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b"} err="failed to get container status \"a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b\": rpc error: code = NotFound desc = could not find container \"a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b\": container with ID starting with a77971ced4cade88a7039bf6721a3ae40696be28c99f12287f6c53f71034258b not found: ID does not exist" Nov 03 07:40:10.646039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.645992 4925 scope.go:122] "RemoveContainer" containerID="714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57" Nov 03 07:40:10.656327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.656304 4925 scope.go:122] "RemoveContainer" containerID="714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57" Nov 03 07:40:10.656637 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:40:10.656613 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57\": container with ID starting with 714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57 not found: ID does not exist" containerID="714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57" Nov 03 07:40:10.656698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.656652 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57"} err="failed to get container status \"714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57\": rpc error: code = NotFound desc = could not find container \"714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57\": container with ID starting with 714b178b4fe4f174e0a85e40b585e0abbd4ce86d1964c023fd933284b863ae57 not found: ID does not exist" Nov 03 07:40:10.742876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.742810 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-envoy\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.742876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.742853 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c9ddd3-3089-48ca-b64e-b429f63b472b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c9ddd3-3089-48ca-b64e-b429f63b472b-istiod-ca-cert\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.743123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.742899 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-certs\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.743123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.742927 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istiod-ca-cert\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.743123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.742971 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-podinfo\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.743123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743045 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-token\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-token\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.743123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743063 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-socket\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.743123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743089 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-podinfo\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.743461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743202 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-certs" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.743461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743353 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c9ddd3-3089-48ca-b64e-b429f63b472b-istiod-ca-cert" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:10.743557 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743385 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istiod-ca-cert" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:10.743557 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743455 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-socket" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.743557 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743480 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-data\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.743557 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743536 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-kube-api-access-5x8cn\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-kube-api-access-5x8cn\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.743723 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743615 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-credential-socket\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.743723 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743671 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-data" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.743723 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743700 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-token\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-token\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743791 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-socket\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743858 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-envoy\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743907 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-envoy\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743959 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80af35f-cccf-496a-a7cb-177e3bc1f399-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80af35f-cccf-496a-a7cb-177e3bc1f399-istiod-ca-cert\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.743987 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-token\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-token\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.744041 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-kube-api-access-wbltq\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-kube-api-access-wbltq\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.744718 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-credential-socket" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.744104 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-kube-api-access-8wskl\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-kube-api-access-8wskl\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745045 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-certs\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745088 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-socket" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745103 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-podinfo\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745138 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-socket\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745206 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-data\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745359 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-certs" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745238 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-certs\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745847 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-credential-socket\") pod \"a80af35f-cccf-496a-a7cb-177e3bc1f399\" (UID: \"a80af35f-cccf-496a-a7cb-177e3bc1f399\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745879 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-data\") pod \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\" (UID: \"2fc42f4e-2481-402d-84c1-7a7e3f81f105\") " Nov 03 07:40:10.746560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.745911 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-credential-socket\") pod \"76c9ddd3-3089-48ca-b64e-b429f63b472b\" (UID: \"76c9ddd3-3089-48ca-b64e-b429f63b472b\") " Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747623 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747652 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c9ddd3-3089-48ca-b64e-b429f63b472b-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747665 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747679 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747694 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747709 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747722 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747750 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-envoy" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747754 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80af35f-cccf-496a-a7cb-177e3bc1f399-istiod-ca-cert" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747736 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747800 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747817 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747831 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss262\" (UniqueName: \"kubernetes.io/projected/6452b9bc-2048-4801-b00a-9e680d27ed5b-kube-api-access-ss262\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747851 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-token" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747867 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6452b9bc-2048-4801-b00a-9e680d27ed5b-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.748063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.747991 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-podinfo" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:10.749161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.748496 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-data" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.749161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.748799 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-credential-socket" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.749161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.748968 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-credential-socket" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.749161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.748992 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-certs" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.749161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.749070 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-data" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.749161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.749116 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-socket" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.749981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.749848 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-envoy" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.749981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.749946 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-podinfo" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:10.750170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.750140 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-token" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.750781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.750753 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-kube-api-access-5x8cn" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "kube-api-access-5x8cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.750895 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.750864 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-envoy" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.751250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.751217 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-token" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.751473 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.751448 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-kube-api-access-wbltq" pod "a80af35f-cccf-496a-a7cb-177e3bc1f399" (UID: "a80af35f-cccf-496a-a7cb-177e3bc1f399"). InnerVolumeSpecName "kube-api-access-wbltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.751773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.751749 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-kube-api-access-8wskl" pod "76c9ddd3-3089-48ca-b64e-b429f63b472b" (UID: "76c9ddd3-3089-48ca-b64e-b429f63b472b"). InnerVolumeSpecName "kube-api-access-8wskl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.752232 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.752209 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-podinfo" pod "2fc42f4e-2481-402d-84c1-7a7e3f81f105" (UID: "2fc42f4e-2481-402d-84c1-7a7e3f81f105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:10.849031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.848976 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849005 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849041 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849050 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80af35f-cccf-496a-a7cb-177e3bc1f399-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849059 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849068 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbltq\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-kube-api-access-wbltq\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849079 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wskl\" (UniqueName: \"kubernetes.io/projected/76c9ddd3-3089-48ca-b64e-b429f63b472b-kube-api-access-8wskl\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849087 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849096 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849105 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849113 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849121 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80af35f-cccf-496a-a7cb-177e3bc1f399-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849129 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849138 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c9ddd3-3089-48ca-b64e-b429f63b472b-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849162 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc42f4e-2481-402d-84c1-7a7e3f81f105-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849171 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849179 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80af35f-cccf-496a-a7cb-177e3bc1f399-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849188 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c9ddd3-3089-48ca-b64e-b429f63b472b-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.849267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:10.849197 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x8cn\" (UniqueName: \"kubernetes.io/projected/2fc42f4e-2481-402d-84c1-7a7e3f81f105-kube-api-access-5x8cn\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:11.302931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.302894 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl"] Nov 03 07:40:11.310091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.310063 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl"] Nov 03 07:40:11.423718 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.423684 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml"] Nov 03 07:40:11.429961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.429931 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml"] Nov 03 07:40:11.481486 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.481396 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:11.544039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.543994 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv"] Nov 03 07:40:11.550084 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.550055 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv"] Nov 03 07:40:11.635092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.635064 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" Nov 03 07:40:11.635521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.635063 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth" event={"ID":"76c9ddd3-3089-48ca-b64e-b429f63b472b","Type":"ContainerDied","Data":"ce61e682cedf842285ef4b45c286d1648c95d71ac92971c2f6e7730aa4966e40"} Nov 03 07:40:11.635521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.635203 4925 scope.go:122] "RemoveContainer" containerID="0bf9af95ec884d6871f22fe2f6cc9981acf2fd33673a81b7cfeb09f4e91fe67e" Nov 03 07:40:11.675303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.675280 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:11.809710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.809631 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth"] Nov 03 07:40:11.816572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:11.816545 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth"] Nov 03 07:40:12.191214 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:12.191172 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" path="/var/lib/kubelet/pods/2fc42f4e-2481-402d-84c1-7a7e3f81f105/volumes" Nov 03 07:40:12.191657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:12.191639 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" path="/var/lib/kubelet/pods/6452b9bc-2048-4801-b00a-9e680d27ed5b/volumes" Nov 03 07:40:12.192732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:12.192606 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" path="/var/lib/kubelet/pods/76c9ddd3-3089-48ca-b64e-b429f63b472b/volumes" Nov 03 07:40:12.193619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:12.193598 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" path="/var/lib/kubelet/pods/a80af35f-cccf-496a-a7cb-177e3bc1f399/volumes" Nov 03 07:40:12.311499 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:12.311467 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:13.904206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:13.904156 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:14.173852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:14.173760 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:14.367096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:14.367054 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:14.803993 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:14.803959 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:19.680637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:19.680600 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:22.254828 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:22.254790 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:22.442293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:22.442256 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:22.566433 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:22.566396 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g"] Nov 03 07:40:22.566702 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:22.566666 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" containerID="cri-o://122f67311791a48cc2df0cdad077977ef2bbd87a9bb8e0c7db64787c49008877" gracePeriod=30 Nov 03 07:40:22.639703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:22.639616 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:25.045860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:25.045820 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:25.948332 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:25.948294 4925 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:40:25.948532 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:25.948373 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:40:26.528073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:26.528034 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:40:27.756066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.756035 4925 generic.go:356] "Generic (PLEG): container finished" podID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerID="122f67311791a48cc2df0cdad077977ef2bbd87a9bb8e0c7db64787c49008877" exitCode=0 Nov 03 07:40:27.756446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.756104 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" event={"ID":"a3a8a718-f328-40ef-83e1-595a8cbba002","Type":"ContainerDied","Data":"122f67311791a48cc2df0cdad077977ef2bbd87a9bb8e0c7db64787c49008877"} Nov 03 07:40:27.846667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.846611 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:40:27.850887 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.850865 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-certs\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.851000 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.850951 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-data\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.851087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851073 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-socket\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.851149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851106 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-token\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-token\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.851149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851114 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-certs" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.851281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851242 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-data" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.851342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851323 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-socket" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.851578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851560 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.851578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851578 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.851698 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.851588 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.853189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.853170 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-token" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:27.952149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.952120 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-credential-socket\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.952310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.952176 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-podinfo\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.952310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.952239 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a8a718-f328-40ef-83e1-595a8cbba002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a8a718-f328-40ef-83e1-595a8cbba002-istiod-ca-cert\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.952310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.952295 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-envoy\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.952483 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.952355 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-kube-api-access-8m6j2\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-kube-api-access-8m6j2\") pod \"a3a8a718-f328-40ef-83e1-595a8cbba002\" (UID: \"a3a8a718-f328-40ef-83e1-595a8cbba002\") " Nov 03 07:40:27.952483 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.952361 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-credential-socket" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.952613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.952587 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a8a718-f328-40ef-83e1-595a8cbba002-istiod-ca-cert" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:27.953151 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.953128 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.953331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.953303 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.953458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.953334 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a8a718-f328-40ef-83e1-595a8cbba002-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.954573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.954541 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-envoy" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.954655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.954594 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-podinfo" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:27.954715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:27.954698 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-kube-api-access-8m6j2" pod "a3a8a718-f328-40ef-83e1-595a8cbba002" (UID: "a3a8a718-f328-40ef-83e1-595a8cbba002"). InnerVolumeSpecName "kube-api-access-8m6j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:28.057256 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.057215 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.057506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.057482 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a8a718-f328-40ef-83e1-595a8cbba002-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.057506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.057506 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6j2\" (UniqueName: \"kubernetes.io/projected/a3a8a718-f328-40ef-83e1-595a8cbba002-kube-api-access-8m6j2\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.763631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.763602 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" Nov 03 07:40:28.763631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.763609 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g" event={"ID":"a3a8a718-f328-40ef-83e1-595a8cbba002","Type":"ContainerDied","Data":"1d40a24e38038589aacc7e7da3cab774cb66ea8a235aa17da0968d0168a44626"} Nov 03 07:40:28.764154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.763674 4925 scope.go:122] "RemoveContainer" containerID="122f67311791a48cc2df0cdad077977ef2bbd87a9bb8e0c7db64787c49008877" Nov 03 07:40:28.988033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.987971 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g"] Nov 03 07:40:28.997240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:28.997215 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g"] Nov 03 07:40:30.154754 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:30.154711 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:30.191236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:30.191197 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" path="/var/lib/kubelet/pods/a3a8a718-f328-40ef-83e1-595a8cbba002/volumes" Nov 03 07:40:35.095295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:35.095264 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:35.338613 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:35.338553 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:39.140041 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:40:39.139961 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:40:40.287497 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:40.287303 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:45.166087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:45.166039 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:45.413322 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:45.413277 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:46.253654 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:46.253615 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:50.337763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:50.337718 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:55.040453 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.040415 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn"] Nov 03 07:40:55.041221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041200 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041222 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041235 4925 cpu_manager.go:414] "containerMap: removing container" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041242 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041256 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041264 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041276 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041284 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" Nov 03 07:40:55.041326 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041329 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041336 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041351 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041358 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041485 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6452b9bc-2048-4801-b00a-9e680d27ed5b" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041498 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80af35f-cccf-496a-a7cb-177e3bc1f399" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041515 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc42f4e-2481-402d-84c1-7a7e3f81f105" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041527 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c9ddd3-3089-48ca-b64e-b429f63b472b" containerName="istio-proxy" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041537 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a870ab4-1945-477a-b018-1fe6fe7f1047" containerName="main" Nov 03 07:40:55.041720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.041548 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a8a718-f328-40ef-83e1-595a8cbba002" containerName="istio-proxy" Nov 03 07:40:55.046504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.046484 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072479 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072450 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072595 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072498 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072595 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072571 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072696 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072760 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072734 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072848 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072873 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b854j\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-kube-api-access-b854j\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.072935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072902 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.073006 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.072990 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.092873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.092849 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57"] Nov 03 07:40:55.096679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.096662 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.097280 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.097260 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h"] Nov 03 07:40:55.101733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.101715 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.138769 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.138742 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89ggc\"" Nov 03 07:40:55.146883 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.146856 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn"] Nov 03 07:40:55.173801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.173768 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.173958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.173808 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.173958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.173843 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174037 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.174114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174096 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174220 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174147 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.174220 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174173 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6802fb28-f1af-4a5a-b353-eb333060b469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.174323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174217 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174239 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.174323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174285 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.174477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174360 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174395 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174423 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.174633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174475 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.174633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174507 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.174633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174540 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.174633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174627 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174654 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174694 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b854j\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-kube-api-access-b854j\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174725 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.174991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174850 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174924 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.174991 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174957 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.175228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.174992 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2xk\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-kube-api-access-7h2xk\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.175228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175065 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6802fb28-f1af-4a5a-b353-eb333060b469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.175228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175094 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.175228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175122 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.175228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175155 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.175228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175191 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.175228 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175208 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.175576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175233 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5kv\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-kube-api-access-pp5kv\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.175576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.175500 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.176961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.176932 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.178111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.178083 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.209045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.201513 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h"] Nov 03 07:40:55.209045 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.208248 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57"] Nov 03 07:40:55.239246 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.239212 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b854j\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-kube-api-access-b854j\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.239643 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.239621 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-qbnpn\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.276534 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276499 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.276534 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276533 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6802fb28-f1af-4a5a-b353-eb333060b469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.276786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276570 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.276786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276597 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.276786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276643 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.276786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276696 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.276786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276736 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.276786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276767 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276852 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276913 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2xk\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-kube-api-access-7h2xk\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276966 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6802fb28-f1af-4a5a-b353-eb333060b469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.276995 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.277040 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.277060 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.277071 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.277154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.277150 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.277556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.277227 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.277556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.277387 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.277556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.277488 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6802fb28-f1af-4a5a-b353-eb333060b469-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278056 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278110 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278112 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5kv\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-kube-api-access-pp5kv\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278165 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278189 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278244 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278467 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.278521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278485 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.278999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.278951 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.280466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.280440 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.280918 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.280883 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.281840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.281809 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6802fb28-f1af-4a5a-b353-eb333060b469-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.282434 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.282416 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.341407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.341377 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.341807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.341786 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5kv\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-kube-api-access-pp5kv\") pod \"maas-default-gateway-openshift-default-58d898bd55-5pm4h\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.341955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.341934 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2xk\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-kube-api-access-7h2xk\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.342031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.341992 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8xt57\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.359867 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.359814 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:55.369072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.369023 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:55.411715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.411670 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:55.417071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.417045 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:55.804418 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.804344 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn"] Nov 03 07:40:55.816178 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.816143 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k"] Nov 03 07:40:55.821093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.821071 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.845636 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:40:55.845607 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda571841e_f2f4_414a_a6b4_cb3d40f0ad29.slice/crio-3a144e90301492f6f892f9bd7f06cad51d4fff3969a31998d99c4486ba78750e WatchSource:0}: Error finding container 3a144e90301492f6f892f9bd7f06cad51d4fff3969a31998d99c4486ba78750e: Status 404 returned error can't find the container with id 3a144e90301492f6f892f9bd7f06cad51d4fff3969a31998d99c4486ba78750e Nov 03 07:40:55.848766 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.848736 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.848874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.848816 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.848874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.848863 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.873477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.873444 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57"] Nov 03 07:40:55.873878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.873795 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.873878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.873847 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.873878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.873876 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.877364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.877320 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0e8bf4\"/\"kube-root-ca.crt\"" Nov 03 07:40:55.877472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.877363 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0e8bf4\"/\"default-dockercfg-zh8bz\"" Nov 03 07:40:55.877472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.877376 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0e8bf4\"/\"openshift-service-ca.crt\"" Nov 03 07:40:55.877472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.877446 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0e8bf4\"/\"test-model-0e8bf4-kserve-self-signed-certs\"" Nov 03 07:40:55.879001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.878977 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k"] Nov 03 07:40:55.884227 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.884208 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tmp-dir\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.884363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.884244 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-model-cache\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.884431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.884387 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-dshm\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.884431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.884423 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-home\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.884550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.884528 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tls-certs\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.884657 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.884634 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7g6\" (UniqueName: \"kubernetes.io/projected/4506a9e5-97e5-48d6-b3bb-20ca29482c37-kube-api-access-5w7g6\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.913872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.913842 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h"] Nov 03 07:40:55.918190 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:40:55.918147 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6802fb28_f1af_4a5a_b353_eb333060b469.slice/crio-4e6023db45cfb9e5ab9683998570dc81b14b9c7dc075023bee7161525dee0796 WatchSource:0}: Error finding container 4e6023db45cfb9e5ab9683998570dc81b14b9c7dc075023bee7161525dee0796: Status 404 returned error can't find the container with id 4e6023db45cfb9e5ab9683998570dc81b14b9c7dc075023bee7161525dee0796 Nov 03 07:40:55.921772 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.921730 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.921857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.921825 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.921897 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.921870 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:40:55.956752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.956683 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:55.966970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.966893 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" event={"ID":"7f65cfe2-7f4e-40d9-9fec-96b82223194c","Type":"ContainerStarted","Data":"7f5424d466a50d7ac6a2a9ec22b0ed40e62228f5cc185ec7a2c485c1c7579ad8"} Nov 03 07:40:55.972930 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.972875 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" event={"ID":"a571841e-f2f4-414a-a6b4-cb3d40f0ad29","Type":"ContainerStarted","Data":"5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4"} Nov 03 07:40:55.976500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.973203 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" event={"ID":"a571841e-f2f4-414a-a6b4-cb3d40f0ad29","Type":"ContainerStarted","Data":"3a144e90301492f6f892f9bd7f06cad51d4fff3969a31998d99c4486ba78750e"} Nov 03 07:40:55.980204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.980169 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" event={"ID":"6802fb28-f1af-4a5a-b353-eb333060b469","Type":"ContainerStarted","Data":"4e6023db45cfb9e5ab9683998570dc81b14b9c7dc075023bee7161525dee0796"} Nov 03 07:40:55.987770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.986428 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7g6\" (UniqueName: \"kubernetes.io/projected/4506a9e5-97e5-48d6-b3bb-20ca29482c37-kube-api-access-5w7g6\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.987770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.986513 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tmp-dir\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.987770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.986556 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-model-cache\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.987770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.986680 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-dshm\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.987770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.986706 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-home\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.987770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.986769 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tls-certs\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.988398 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.988372 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tmp-dir\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.988577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.988560 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-model-cache\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.988968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.988947 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-home\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.990547 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.990525 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-dshm\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:55.992140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:55.992116 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tls-certs\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:56.052736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:56.052672 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7g6\" (UniqueName: \"kubernetes.io/projected/4506a9e5-97e5-48d6-b3bb-20ca29482c37-kube-api-access-5w7g6\") pod \"test-model-0e8bf4-kserve-859fd7bbb5-vsg7k\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:56.106328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:56.106283 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podStartSLOduration=2.106268775 podStartE2EDuration="2.106268775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:56.103064268 +0000 UTC m=+2516.036592926" watchObservedRunningTime="2026-11-03 07:40:56.106268775 +0000 UTC m=+2516.039797430" Nov 03 07:40:56.139434 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:56.138994 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:56.366675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:56.366467 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:40:56.368990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:56.368727 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:40:56.368990 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:56.368781 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:40:56.602482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:56.602447 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k"] Nov 03 07:40:57.017035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.010697 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" event={"ID":"6802fb28-f1af-4a5a-b353-eb333060b469","Type":"ContainerStarted","Data":"3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084"} Nov 03 07:40:57.017035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.014720 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" event={"ID":"4506a9e5-97e5-48d6-b3bb-20ca29482c37","Type":"ContainerStarted","Data":"44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b"} Nov 03 07:40:57.017035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.014769 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" event={"ID":"4506a9e5-97e5-48d6-b3bb-20ca29482c37","Type":"ContainerStarted","Data":"ee5b1ac4a3f0db24484113d4fc90e857f0b3689b1a1538c2c1f83020a2daa2f3"} Nov 03 07:40:57.022035 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.019242 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" event={"ID":"7f65cfe2-7f4e-40d9-9fec-96b82223194c","Type":"ContainerStarted","Data":"6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e"} Nov 03 07:40:57.140125 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.140093 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:40:57.176039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.173148 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podStartSLOduration=2.173125681 podStartE2EDuration="2.173125681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:57.171614239 +0000 UTC m=+2517.105142939" watchObservedRunningTime="2026-11-03 07:40:57.173125681 +0000 UTC m=+2517.106654340" Nov 03 07:40:57.202665 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.202518 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:40:57.313039 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.308622 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" podStartSLOduration=2.308597526 podStartE2EDuration="2.308597526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:57.294371437 +0000 UTC m=+2517.227900095" watchObservedRunningTime="2026-11-03 07:40:57.308597526 +0000 UTC m=+2517.242126185" Nov 03 07:40:57.381037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.380438 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:40:57.381037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.380501 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:40:57.439505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.439471 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:40:57.439801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.439782 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:40:57.439962 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.439938 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:40:57.441494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.441470 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:40:57.441792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.441740 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:40:57.463094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:57.450298 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:40:58.364477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:58.364438 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:40:58.365161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:58.365121 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:40:58.425666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:58.425619 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:40:58.425871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:58.425688 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:40:58.425871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:58.425762 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:40:58.425871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:58.425781 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:40:59.364068 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:59.364033 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:40:59.364775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:59.364307 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:40:59.414540 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:59.414504 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:40:59.414855 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:59.414812 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:40:59.419436 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:59.419394 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:40:59.419646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:40:59.419623 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:41:00.372037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:00.368052 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:41:00.372037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:00.368113 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:41:00.415662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:00.415442 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:41:00.415662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:00.415506 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:41:00.420491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:00.420380 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:41:00.420491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:00.420429 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:41:00.763095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:00.712538 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:01.041685 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.041519 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:01.256827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.256777 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:01.362268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.362235 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:41:01.362620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.362589 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:41:01.413460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.413423 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:41:01.414084 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.414050 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:41:01.422095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.422068 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:41:01.422888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:01.422859 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:41:02.363040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:02.362402 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:41:02.363040 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:02.362466 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:41:02.412561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:02.412518 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:41:02.412845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:02.412813 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:41:02.418354 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:02.418331 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:41:02.418741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:02.418716 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:41:03.362154 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:03.362111 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 07:41:03.362489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:03.362452 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 07:41:03.412315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:03.412276 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:41:03.412616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:03.412581 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:41:03.418871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:03.418838 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:41:03.419324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:03.418898 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:41:04.412382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:04.412353 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 07:41:04.412636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:04.412608 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 07:41:04.418471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:04.418437 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 07:41:04.418590 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:04.418508 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 07:41:05.122754 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.122709 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:05.340208 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.340173 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:05.405384 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.405262 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:41:05.405672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.405624 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:41:05.660092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.659991 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:41:05.661874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.661844 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:41:05.665311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.665285 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:41:05.830176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.830141 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:41:05.830834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.830769 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:41:05.832290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.832240 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:41:05.839303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.839256 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podStartSLOduration=10.83924359 podStartE2EDuration="10.83924359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:57.44206721 +0000 UTC m=+2517.375595866" watchObservedRunningTime="2026-11-03 07:41:05.83924359 +0000 UTC m=+2525.772772246" Nov 03 07:41:05.892276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.892235 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:41:05.892703 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.892662 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:41:05.893777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:05.893758 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:41:06.140374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:06.139677 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:41:06.152800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:06.152769 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:41:06.258858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:06.258831 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:41:06.352358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:06.352322 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:09.139585 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:41:09.139548 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:41:11.338254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:11.338206 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:14.475243 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:14.475200 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:14.679072 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:14.679002 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:16.438662 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:16.438616 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:17.558582 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:17.558550 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:41:19.380249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:19.380192 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t"] Nov 03 07:41:19.430890 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:19.430850 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk"] Nov 03 07:41:19.434126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:19.434090 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7"] Nov 03 07:41:19.594491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:19.594457 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:21.873626 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:21.873596 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:26.510491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.510454 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl"] Nov 03 07:41:26.511095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.510680 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" podUID="b61eccd2-6a03-4dd9-829f-414965b61fa1" containerName="maas-api" containerID="cri-o://7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c" gracePeriod=30 Nov 03 07:41:26.665293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.665252 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:26.855066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.849911 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:26.873080 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.861368 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:41:26.966806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.966001 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61eccd2-6a03-4dd9-829f-414965b61fa1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61eccd2-6a03-4dd9-829f-414965b61fa1-openshift-service-ca\") pod \"b61eccd2-6a03-4dd9-829f-414965b61fa1\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " Nov 03 07:41:26.966806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.966093 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls\") pod \"b61eccd2-6a03-4dd9-829f-414965b61fa1\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " Nov 03 07:41:26.966806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.966185 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-metrics-certs\") pod \"b61eccd2-6a03-4dd9-829f-414965b61fa1\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " Nov 03 07:41:26.966806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.966240 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61eccd2-6a03-4dd9-829f-414965b61fa1-kube-api-access-ffmtw\" (UniqueName: \"kubernetes.io/projected/b61eccd2-6a03-4dd9-829f-414965b61fa1-kube-api-access-ffmtw\") pod \"b61eccd2-6a03-4dd9-829f-414965b61fa1\" (UID: \"b61eccd2-6a03-4dd9-829f-414965b61fa1\") " Nov 03 07:41:26.966806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.966378 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61eccd2-6a03-4dd9-829f-414965b61fa1-openshift-service-ca" pod "b61eccd2-6a03-4dd9-829f-414965b61fa1" (UID: "b61eccd2-6a03-4dd9-829f-414965b61fa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:41:26.967244 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.966933 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61eccd2-6a03-4dd9-829f-414965b61fa1-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:26.989922 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.970414 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-metrics-certs" pod "b61eccd2-6a03-4dd9-829f-414965b61fa1" (UID: "b61eccd2-6a03-4dd9-829f-414965b61fa1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:41:26.989922 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.970648 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls" pod "b61eccd2-6a03-4dd9-829f-414965b61fa1" (UID: "b61eccd2-6a03-4dd9-829f-414965b61fa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:41:26.989922 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:26.970757 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61eccd2-6a03-4dd9-829f-414965b61fa1-kube-api-access-ffmtw" pod "b61eccd2-6a03-4dd9-829f-414965b61fa1" (UID: "b61eccd2-6a03-4dd9-829f-414965b61fa1"). InnerVolumeSpecName "kube-api-access-ffmtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:27.058134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.058095 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:27.068886 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.068858 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:27.068886 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.068892 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffmtw\" (UniqueName: \"kubernetes.io/projected/b61eccd2-6a03-4dd9-829f-414965b61fa1-kube-api-access-ffmtw\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:27.069073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.068908 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b61eccd2-6a03-4dd9-829f-414965b61fa1-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:27.426925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.426886 4925 generic.go:356] "Generic (PLEG): container finished" podID="b61eccd2-6a03-4dd9-829f-414965b61fa1" containerID="7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c" exitCode=0 Nov 03 07:41:27.427130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.426944 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" Nov 03 07:41:27.427130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.426968 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" event={"ID":"b61eccd2-6a03-4dd9-829f-414965b61fa1","Type":"ContainerDied","Data":"7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c"} Nov 03 07:41:27.427130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.427079 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl" event={"ID":"b61eccd2-6a03-4dd9-829f-414965b61fa1","Type":"ContainerDied","Data":"094e78d306eed043e2eb2012d85995feaec1760ecf9c84cda8019817ccdafbd4"} Nov 03 07:41:27.427130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.427111 4925 scope.go:122] "RemoveContainer" containerID="7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c" Nov 03 07:41:27.438745 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.438724 4925 scope.go:122] "RemoveContainer" containerID="7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c" Nov 03 07:41:27.439126 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:41:27.439101 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c\": container with ID starting with 7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c not found: ID does not exist" containerID="7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c" Nov 03 07:41:27.439225 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.439137 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c"} err="failed to get container status \"7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c\": rpc error: code = NotFound desc = could not find container \"7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c\": container with ID starting with 7b2a3f0c4ac6ad210bce02ad6cabf96ba0c1b12227f4daf93d74a6459fb12c4c not found: ID does not exist" Nov 03 07:41:27.608554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.608519 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl"] Nov 03 07:41:27.611692 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:27.611668 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-3b41b9-86b58f78d4-xv7nl"] Nov 03 07:41:28.191201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:28.191160 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61eccd2-6a03-4dd9-829f-414965b61fa1" path="/var/lib/kubelet/pods/b61eccd2-6a03-4dd9-829f-414965b61fa1/volumes" Nov 03 07:41:28.342066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:28.342006 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:28.608811 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:28.608771 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:28.855530 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:28.855497 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:29.386240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:29.386186 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:33.934935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:33.934745 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:34.401257 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.401193 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" containerID="cri-o://6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5" gracePeriod=45 Nov 03 07:41:34.452653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.452616 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" containerID="cri-o://922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf" gracePeriod=45 Nov 03 07:41:34.455222 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.455165 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" containerID="cri-o://b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387" gracePeriod=45 Nov 03 07:41:34.543819 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.543783 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": dial tcp 10.132.0.59:8000: connect: connection refused" Nov 03 07:41:34.752124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.752100 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:41:34.766198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.766172 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:41:34.777874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.777802 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:41:34.875508 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875476 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-model-cache\") pod \"cef01dc3-a11f-461c-8d55-78763b38f002\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " Nov 03 07:41:34.875708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875540 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-dshm\") pod \"cef01dc3-a11f-461c-8d55-78763b38f002\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " Nov 03 07:41:34.875708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875571 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a95df8-8830-4852-af2b-ecf9b484bf3c-kube-api-access-7hgxb\" (UniqueName: \"kubernetes.io/projected/f5a95df8-8830-4852-af2b-ecf9b484bf3c-kube-api-access-7hgxb\") pod \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " Nov 03 07:41:34.875708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875601 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd5a47fd-67ac-479c-8064-b067ea059c6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5a47fd-67ac-479c-8064-b067ea059c6f-tls-certs\") pod \"dd5a47fd-67ac-479c-8064-b067ea059c6f\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " Nov 03 07:41:34.875708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875658 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-home\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-home\") pod \"cef01dc3-a11f-461c-8d55-78763b38f002\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " Nov 03 07:41:34.875708 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875689 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-dshm\") pod \"dd5a47fd-67ac-479c-8064-b067ea059c6f\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " Nov 03 07:41:34.875975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875780 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-tmp-dir\") pod \"cef01dc3-a11f-461c-8d55-78763b38f002\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " Nov 03 07:41:34.875975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875785 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-model-cache" pod "cef01dc3-a11f-461c-8d55-78763b38f002" (UID: "cef01dc3-a11f-461c-8d55-78763b38f002"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.875975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875835 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tmp-dir\") pod \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " Nov 03 07:41:34.875975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875877 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-home\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-home\") pod \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " Nov 03 07:41:34.875975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875931 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cef01dc3-a11f-461c-8d55-78763b38f002-tls-certs\" (UniqueName: \"kubernetes.io/secret/cef01dc3-a11f-461c-8d55-78763b38f002-tls-certs\") pod \"cef01dc3-a11f-461c-8d55-78763b38f002\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " Nov 03 07:41:34.876283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.875972 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-dshm\") pod \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " Nov 03 07:41:34.876283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876043 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-tmp-dir\") pod \"dd5a47fd-67ac-479c-8064-b067ea059c6f\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " Nov 03 07:41:34.876283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876077 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-home\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-home\") pod \"dd5a47fd-67ac-479c-8064-b067ea059c6f\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " Nov 03 07:41:34.876283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876120 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef01dc3-a11f-461c-8d55-78763b38f002-kube-api-access-nqkbh\" (UniqueName: \"kubernetes.io/projected/cef01dc3-a11f-461c-8d55-78763b38f002-kube-api-access-nqkbh\") pod \"cef01dc3-a11f-461c-8d55-78763b38f002\" (UID: \"cef01dc3-a11f-461c-8d55-78763b38f002\") " Nov 03 07:41:34.876283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876161 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tls-certs\") pod \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " Nov 03 07:41:34.876283 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876256 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5a47fd-67ac-479c-8064-b067ea059c6f-kube-api-access-p6zjq\" (UniqueName: \"kubernetes.io/projected/dd5a47fd-67ac-479c-8064-b067ea059c6f-kube-api-access-p6zjq\") pod \"dd5a47fd-67ac-479c-8064-b067ea059c6f\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " Nov 03 07:41:34.876573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876301 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-model-cache\") pod \"dd5a47fd-67ac-479c-8064-b067ea059c6f\" (UID: \"dd5a47fd-67ac-479c-8064-b067ea059c6f\") " Nov 03 07:41:34.876573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876350 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-model-cache\") pod \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\" (UID: \"f5a95df8-8830-4852-af2b-ecf9b484bf3c\") " Nov 03 07:41:34.876857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.876838 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.878416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.877097 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-model-cache" pod "f5a95df8-8830-4852-af2b-ecf9b484bf3c" (UID: "f5a95df8-8830-4852-af2b-ecf9b484bf3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.878416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.877165 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-tmp-dir" pod "dd5a47fd-67ac-479c-8064-b067ea059c6f" (UID: "dd5a47fd-67ac-479c-8064-b067ea059c6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.878416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.877187 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-tmp-dir" pod "cef01dc3-a11f-461c-8d55-78763b38f002" (UID: "cef01dc3-a11f-461c-8d55-78763b38f002"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.878416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.877401 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-home" pod "cef01dc3-a11f-461c-8d55-78763b38f002" (UID: "cef01dc3-a11f-461c-8d55-78763b38f002"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.878416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.877641 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-model-cache" pod "dd5a47fd-67ac-479c-8064-b067ea059c6f" (UID: "dd5a47fd-67ac-479c-8064-b067ea059c6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.878416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.877891 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-home" pod "dd5a47fd-67ac-479c-8064-b067ea059c6f" (UID: "dd5a47fd-67ac-479c-8064-b067ea059c6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.878416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.878113 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-home" pod "f5a95df8-8830-4852-af2b-ecf9b484bf3c" (UID: "f5a95df8-8830-4852-af2b-ecf9b484bf3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.879137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.878520 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tmp-dir" pod "f5a95df8-8830-4852-af2b-ecf9b484bf3c" (UID: "f5a95df8-8830-4852-af2b-ecf9b484bf3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.880680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.880654 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tls-certs" pod "f5a95df8-8830-4852-af2b-ecf9b484bf3c" (UID: "f5a95df8-8830-4852-af2b-ecf9b484bf3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:41:34.881507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.881484 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-dshm" pod "dd5a47fd-67ac-479c-8064-b067ea059c6f" (UID: "dd5a47fd-67ac-479c-8064-b067ea059c6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.882422 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.882330 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-dshm" pod "cef01dc3-a11f-461c-8d55-78763b38f002" (UID: "cef01dc3-a11f-461c-8d55-78763b38f002"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.882422 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.882342 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a95df8-8830-4852-af2b-ecf9b484bf3c-kube-api-access-7hgxb" pod "f5a95df8-8830-4852-af2b-ecf9b484bf3c" (UID: "f5a95df8-8830-4852-af2b-ecf9b484bf3c"). InnerVolumeSpecName "kube-api-access-7hgxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:34.882422 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.882385 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-dshm" pod "f5a95df8-8830-4852-af2b-ecf9b484bf3c" (UID: "f5a95df8-8830-4852-af2b-ecf9b484bf3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.883031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.882984 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef01dc3-a11f-461c-8d55-78763b38f002-kube-api-access-nqkbh" pod "cef01dc3-a11f-461c-8d55-78763b38f002" (UID: "cef01dc3-a11f-461c-8d55-78763b38f002"). InnerVolumeSpecName "kube-api-access-nqkbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:34.883196 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.883177 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5a47fd-67ac-479c-8064-b067ea059c6f-kube-api-access-p6zjq" pod "dd5a47fd-67ac-479c-8064-b067ea059c6f" (UID: "dd5a47fd-67ac-479c-8064-b067ea059c6f"). InnerVolumeSpecName "kube-api-access-p6zjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:34.883936 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.883898 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd5a47fd-67ac-479c-8064-b067ea059c6f-tls-certs" pod "dd5a47fd-67ac-479c-8064-b067ea059c6f" (UID: "dd5a47fd-67ac-479c-8064-b067ea059c6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:41:34.884132 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.884110 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef01dc3-a11f-461c-8d55-78763b38f002-tls-certs" pod "cef01dc3-a11f-461c-8d55-78763b38f002" (UID: "cef01dc3-a11f-461c-8d55-78763b38f002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977739 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977772 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef01dc3-a11f-461c-8d55-78763b38f002-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977783 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977795 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977808 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977818 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqkbh\" (UniqueName: \"kubernetes.io/projected/cef01dc3-a11f-461c-8d55-78763b38f002-kube-api-access-nqkbh\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977828 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.977835 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977836 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6zjq\" (UniqueName: \"kubernetes.io/projected/dd5a47fd-67ac-479c-8064-b067ea059c6f-kube-api-access-p6zjq\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977846 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977854 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977862 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977872 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hgxb\" (UniqueName: \"kubernetes.io/projected/f5a95df8-8830-4852-af2b-ecf9b484bf3c-kube-api-access-7hgxb\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977885 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd5a47fd-67ac-479c-8064-b067ea059c6f-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977894 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977902 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd5a47fd-67ac-479c-8064-b067ea059c6f-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977910 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef01dc3-a11f-461c-8d55-78763b38f002-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.978738 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:34.977917 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a95df8-8830-4852-af2b-ecf9b484bf3c-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:35.495102 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.495072 4925 generic.go:356] "Generic (PLEG): container finished" podID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerID="b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387" exitCode=0 Nov 03 07:41:35.495258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.495121 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" Nov 03 07:41:35.495258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.495166 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" event={"ID":"dd5a47fd-67ac-479c-8064-b067ea059c6f","Type":"ContainerDied","Data":"b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387"} Nov 03 07:41:35.495258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.495211 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7" event={"ID":"dd5a47fd-67ac-479c-8064-b067ea059c6f","Type":"ContainerDied","Data":"d94bde3314ee37828b8a2ce726982af9fdec29e306b9144d56fa1e1d1ab32198"} Nov 03 07:41:35.495258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.495237 4925 scope.go:122] "RemoveContainer" containerID="b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387" Nov 03 07:41:35.496756 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.496714 4925 generic.go:356] "Generic (PLEG): container finished" podID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerID="6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5" exitCode=0 Nov 03 07:41:35.496882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.496762 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" Nov 03 07:41:35.497061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.497042 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" event={"ID":"f5a95df8-8830-4852-af2b-ecf9b484bf3c","Type":"ContainerDied","Data":"6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5"} Nov 03 07:41:35.497139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.497080 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t" event={"ID":"f5a95df8-8830-4852-af2b-ecf9b484bf3c","Type":"ContainerDied","Data":"d0a9e35b4fb555201f91f6a7fb98a51c64cf27b50be811138f64bc0f32d63cee"} Nov 03 07:41:35.498761 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.498736 4925 generic.go:356] "Generic (PLEG): container finished" podID="cef01dc3-a11f-461c-8d55-78763b38f002" containerID="922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf" exitCode=0 Nov 03 07:41:35.498880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.498766 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" event={"ID":"cef01dc3-a11f-461c-8d55-78763b38f002","Type":"ContainerDied","Data":"922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf"} Nov 03 07:41:35.498880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.498791 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" Nov 03 07:41:35.498880 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.498795 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk" event={"ID":"cef01dc3-a11f-461c-8d55-78763b38f002","Type":"ContainerDied","Data":"24760eab9e1b276671dcb83204bebd2a19831b617d2c59f15ab8551842764fe8"} Nov 03 07:41:35.508261 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.508227 4925 scope.go:122] "RemoveContainer" containerID="b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387" Nov 03 07:41:35.509357 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:41:35.508573 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387\": container with ID starting with b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387 not found: ID does not exist" containerID="b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387" Nov 03 07:41:35.509357 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.508961 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387"} err="failed to get container status \"b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387\": rpc error: code = NotFound desc = could not find container \"b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387\": container with ID starting with b383366bc50ceec7a37860145f49a9671a6f643d1eaa3eed564a64acd25c3387 not found: ID does not exist" Nov 03 07:41:35.509357 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.509094 4925 scope.go:122] "RemoveContainer" containerID="6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5" Nov 03 07:41:35.523870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.523848 4925 scope.go:122] "RemoveContainer" containerID="6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5" Nov 03 07:41:35.524163 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:41:35.524141 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5\": container with ID starting with 6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5 not found: ID does not exist" containerID="6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5" Nov 03 07:41:35.524217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.524172 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5"} err="failed to get container status \"6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5\": rpc error: code = NotFound desc = could not find container \"6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5\": container with ID starting with 6ebfb217ee5f428d07f5beda4b71376d831959161cf1f1b9059bdf07634e05d5 not found: ID does not exist" Nov 03 07:41:35.524217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.524188 4925 scope.go:122] "RemoveContainer" containerID="922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf" Nov 03 07:41:35.533619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.533600 4925 scope.go:122] "RemoveContainer" containerID="922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf" Nov 03 07:41:35.533871 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:41:35.533847 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf\": container with ID starting with 922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf not found: ID does not exist" containerID="922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf" Nov 03 07:41:35.533938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.533878 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf"} err="failed to get container status \"922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf\": rpc error: code = NotFound desc = could not find container \"922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf\": container with ID starting with 922506d418c2b88fa8fc65eebc699f3bb117cd1d8b4b1cb962fc6bbe0f33facf not found: ID does not exist" Nov 03 07:41:35.667679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.667644 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk"] Nov 03 07:41:35.717970 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.717891 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/premium-w1-9697f7-kserve-bdd477bc7-kw6tk"] Nov 03 07:41:35.791603 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.791569 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t"] Nov 03 07:41:35.797355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.797330 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-2-w1-9697f7-kserve-7ff4d4cf7f-k468t"] Nov 03 07:41:35.911199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.911165 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7"] Nov 03 07:41:35.916798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:35.916768 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/simulator-w1-9697f7-kserve-5f6dcd8496-w5tj7"] Nov 03 07:41:36.169998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.169963 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:36.200625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.200587 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" path="/var/lib/kubelet/pods/cef01dc3-a11f-461c-8d55-78763b38f002/volumes" Nov 03 07:41:36.201398 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.201373 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" path="/var/lib/kubelet/pods/dd5a47fd-67ac-479c-8064-b067ea059c6f/volumes" Nov 03 07:41:36.202047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.202024 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" path="/var/lib/kubelet/pods/f5a95df8-8830-4852-af2b-ecf9b484bf3c/volumes" Nov 03 07:41:36.418287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.418252 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:36.477122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.472907 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg"] Nov 03 07:41:36.477122 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.473223 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" containerID="cri-o://af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be" gracePeriod=30 Nov 03 07:41:36.603910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:36.603638 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:39.139125 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:41:39.139084 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:41:39.641441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:39.641400 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:40.987667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:40.987586 4925 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:41:40.987667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:40.987655 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:41:41.105928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.105892 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:41.296377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.296301 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:41.747646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.747613 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:41:41.761469 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761439 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-certs\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761473 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-token\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-token\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761517 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-kube-api-access-bj7c2\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-kube-api-access-bj7c2\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761540 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-data\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761621 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-podinfo\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761663 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a589ad64-54f2-4ae8-8844-351469ef11f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589ad64-54f2-4ae8-8844-351469ef11f2-istiod-ca-cert\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761703 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-socket\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761730 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-credential-socket\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761995 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761809 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-envoy\") pod \"a589ad64-54f2-4ae8-8844-351469ef11f2\" (UID: \"a589ad64-54f2-4ae8-8844-351469ef11f2\") " Nov 03 07:41:41.761995 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.761698 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-certs" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.762139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762109 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-socket" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.762139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762131 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a589ad64-54f2-4ae8-8844-351469ef11f2-istiod-ca-cert" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:41:41.762249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762126 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-credential-socket" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.762249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762196 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-data" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.762441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762383 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.762441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762416 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.762441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762431 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589ad64-54f2-4ae8-8844-351469ef11f2-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.762441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762445 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.762441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.762459 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.764119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.764087 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-kube-api-access-bj7c2" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "kube-api-access-bj7c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:41.764119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.764095 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-envoy" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.764286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.764162 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-podinfo" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:41:41.764286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.764251 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-token" pod "a589ad64-54f2-4ae8-8844-351469ef11f2" (UID: "a589ad64-54f2-4ae8-8844-351469ef11f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:41.863794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.863711 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.863794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.863747 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7c2\" (UniqueName: \"kubernetes.io/projected/a589ad64-54f2-4ae8-8844-351469ef11f2-kube-api-access-bj7c2\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.863794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.863758 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.863794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:41.863768 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589ad64-54f2-4ae8-8844-351469ef11f2-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:42.551733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.551650 4925 generic.go:356] "Generic (PLEG): container finished" podID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerID="af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be" exitCode=0 Nov 03 07:41:42.551733 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.551705 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" Nov 03 07:41:42.552250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.551737 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" event={"ID":"a589ad64-54f2-4ae8-8844-351469ef11f2","Type":"ContainerDied","Data":"af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be"} Nov 03 07:41:42.552250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.551798 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg" event={"ID":"a589ad64-54f2-4ae8-8844-351469ef11f2","Type":"ContainerDied","Data":"ca39e4b78d91ca51ab356839a2a4f237e33669212e001ef75460f5b16e5a908b"} Nov 03 07:41:42.552250 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.551822 4925 scope.go:122] "RemoveContainer" containerID="af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be" Nov 03 07:41:42.561941 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.561925 4925 scope.go:122] "RemoveContainer" containerID="af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be" Nov 03 07:41:42.562282 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:41:42.562261 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be\": container with ID starting with af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be not found: ID does not exist" containerID="af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be" Nov 03 07:41:42.562374 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.562288 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be"} err="failed to get container status \"af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be\": rpc error: code = NotFound desc = could not find container \"af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be\": container with ID starting with af171b38f980c28fe823e34a7c32e69b45319680ea1446e48dd1c92df0f826be not found: ID does not exist" Nov 03 07:41:42.724137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.724104 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg"] Nov 03 07:41:42.774667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:42.774634 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg"] Nov 03 07:41:44.191533 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:44.191489 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" path="/var/lib/kubelet/pods/a589ad64-54f2-4ae8-8844-351469ef11f2/volumes" Nov 03 07:41:44.704522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:44.704468 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:46.932132 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:46.932092 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:49.734208 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:49.734165 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:52.941195 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:52.941156 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:54.209795 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:54.209764 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:41:55.033998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:55.033955 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:57.176606 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:57.176569 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:57.413973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:57.413878 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:41:59.306814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:41:59.306781 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:00.463730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:00.463686 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:04.947604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:04.947565 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:05.443276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:05.443235 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:05.745186 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:05.745080 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:06.860071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:06.860032 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:08.881231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:08.881194 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:09.084435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:09.084393 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:09.139195 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:42:09.139117 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:42:11.109873 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:11.109826 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:13.684722 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:13.684680 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:13.904315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:13.904277 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:16.223855 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:16.223807 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:17.361380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:17.361343 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:18.029106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:18.029070 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7"] Nov 03 07:42:18.139296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:18.139259 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54"] Nov 03 07:42:18.454585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:18.454551 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:18.681071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:18.680979 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:21.616700 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:21.616645 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:23.030265 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.030225 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" podUID="be964d3c-36c1-43fd-b8da-b5483f6819ac" containerName="payload-processing" containerID="cri-o://f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604" gracePeriod=25 Nov 03 07:42:23.140489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.140433 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" podUID="f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" containerName="payload-pre-processing" containerID="cri-o://23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87" gracePeriod=25 Nov 03 07:42:23.308080 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.308051 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:42:23.387393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.387355 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be964d3c-36c1-43fd-b8da-b5483f6819ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be964d3c-36c1-43fd-b8da-b5483f6819ac-plugins-config-volume\") pod \"be964d3c-36c1-43fd-b8da-b5483f6819ac\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " Nov 03 07:42:23.387601 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.387576 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be964d3c-36c1-43fd-b8da-b5483f6819ac-kube-api-access-r4mnb\" (UniqueName: \"kubernetes.io/projected/be964d3c-36c1-43fd-b8da-b5483f6819ac-kube-api-access-r4mnb\") pod \"be964d3c-36c1-43fd-b8da-b5483f6819ac\" (UID: \"be964d3c-36c1-43fd-b8da-b5483f6819ac\") " Nov 03 07:42:23.388505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.388475 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be964d3c-36c1-43fd-b8da-b5483f6819ac-plugins-config-volume" pod "be964d3c-36c1-43fd-b8da-b5483f6819ac" (UID: "be964d3c-36c1-43fd-b8da-b5483f6819ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:23.391270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.391240 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be964d3c-36c1-43fd-b8da-b5483f6819ac-kube-api-access-r4mnb" pod "be964d3c-36c1-43fd-b8da-b5483f6819ac" (UID: "be964d3c-36c1-43fd-b8da-b5483f6819ac"). InnerVolumeSpecName "kube-api-access-r4mnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:23.413739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.413719 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:42:23.488652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.488617 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-kube-api-access-vl45z\" (UniqueName: \"kubernetes.io/projected/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-kube-api-access-vl45z\") pod \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " Nov 03 07:42:23.488870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.488782 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-plugins-config-volume\") pod \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\" (UID: \"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2\") " Nov 03 07:42:23.489128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.489108 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mnb\" (UniqueName: \"kubernetes.io/projected/be964d3c-36c1-43fd-b8da-b5483f6819ac-kube-api-access-r4mnb\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:23.489128 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.489129 4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be964d3c-36c1-43fd-b8da-b5483f6819ac-plugins-config-volume\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:23.489309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.489186 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-plugins-config-volume" pod "f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" (UID: "f1ae1ea2-c7c3-4427-af01-f73d09e48cd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:23.491517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.491490 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-kube-api-access-vl45z" pod "f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" (UID: "f1ae1ea2-c7c3-4427-af01-f73d09e48cd2"). InnerVolumeSpecName "kube-api-access-vl45z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:23.591142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.591103 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl45z\" (UniqueName: \"kubernetes.io/projected/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-kube-api-access-vl45z\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:23.591142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.591141 4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2-plugins-config-volume\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:23.854234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.854148 4925 generic.go:356] "Generic (PLEG): container finished" podID="f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" containerID="23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87" exitCode=0 Nov 03 07:42:23.854234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.854195 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" Nov 03 07:42:23.854444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.854232 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" event={"ID":"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2","Type":"ContainerDied","Data":"23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87"} Nov 03 07:42:23.854444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.854291 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54" event={"ID":"f1ae1ea2-c7c3-4427-af01-f73d09e48cd2","Type":"ContainerDied","Data":"66eea9de1d49b3768d455692ccfb4551ae49ffa7ba795d36f485b25b5b674a98"} Nov 03 07:42:23.854444 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.854321 4925 scope.go:122] "RemoveContainer" containerID="23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87" Nov 03 07:42:23.856176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.856149 4925 generic.go:356] "Generic (PLEG): container finished" podID="be964d3c-36c1-43fd-b8da-b5483f6819ac" containerID="f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604" exitCode=0 Nov 03 07:42:23.856301 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.856241 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" event={"ID":"be964d3c-36c1-43fd-b8da-b5483f6819ac","Type":"ContainerDied","Data":"f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604"} Nov 03 07:42:23.856301 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.856294 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" event={"ID":"be964d3c-36c1-43fd-b8da-b5483f6819ac","Type":"ContainerDied","Data":"f0ba5d9d9816a40ea8265186b7230367e923d30a2624239d955ccf5d2b3950ff"} Nov 03 07:42:23.856449 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.856431 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7" Nov 03 07:42:23.867862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.867844 4925 scope.go:122] "RemoveContainer" containerID="23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87" Nov 03 07:42:23.868202 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:42:23.868171 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87\": container with ID starting with 23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87 not found: ID does not exist" containerID="23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87" Nov 03 07:42:23.868299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.868210 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87"} err="failed to get container status \"23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87\": rpc error: code = NotFound desc = could not find container \"23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87\": container with ID starting with 23585dd502474ef4d68ebdf3039786401a1813513bc6d1569fc68ab93c93af87 not found: ID does not exist" Nov 03 07:42:23.868299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.868236 4925 scope.go:122] "RemoveContainer" containerID="f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604" Nov 03 07:42:23.879359 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.879322 4925 scope.go:122] "RemoveContainer" containerID="f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604" Nov 03 07:42:23.879655 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:42:23.879623 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604\": container with ID starting with f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604 not found: ID does not exist" containerID="f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604" Nov 03 07:42:23.879757 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:23.879660 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604"} err="failed to get container status \"f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604\": rpc error: code = NotFound desc = could not find container \"f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604\": container with ID starting with f770d00a20d219f1386344324c72c00abed55ae2acaa04236ea5f3773a666604 not found: ID does not exist" Nov 03 07:42:24.027048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:24.026997 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7"] Nov 03 07:42:24.077785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:24.077754 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7"] Nov 03 07:42:24.147452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:24.147373 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54"] Nov 03 07:42:24.151850 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:24.151821 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54"] Nov 03 07:42:24.193046 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:24.192989 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be964d3c-36c1-43fd-b8da-b5483f6819ac" path="/var/lib/kubelet/pods/be964d3c-36c1-43fd-b8da-b5483f6819ac/volumes" Nov 03 07:42:24.193644 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:24.193625 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" path="/var/lib/kubelet/pods/f1ae1ea2-c7c3-4427-af01-f73d09e48cd2/volumes" Nov 03 07:42:26.588767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:26.588727 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:31.536565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:31.536526 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:33.330846 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:33.330813 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k"] Nov 03 07:42:33.331305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:33.331056 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" containerID="cri-o://920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2" gracePeriod=30 Nov 03 07:42:34.078217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:34.078185 4925 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:42:34.078417 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:34.078234 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:42:36.546624 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:36.546586 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:38.592543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.592518 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:42:38.669441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669351 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-envoy\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669441 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669392 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-certs\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669447 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-data\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669473 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-socket\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669573 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-credential-socket\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669610 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-podinfo\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669637 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-token\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-token\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669712 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac500e48-730e-41f4-8521-0c7f53e7687f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac500e48-730e-41f4-8521-0c7f53e7687f-istiod-ca-cert\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669755 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-kube-api-access-fdh5l\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-kube-api-access-fdh5l\") pod \"ac500e48-730e-41f4-8521-0c7f53e7687f\" (UID: \"ac500e48-730e-41f4-8521-0c7f53e7687f\") " Nov 03 07:42:38.669909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669772 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-certs" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:38.670069 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.669934 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-data" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:38.670311 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.670293 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.670353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.670317 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.670655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.670538 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-socket" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:38.670655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.670573 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-credential-socket" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:38.670820 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.670769 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac500e48-730e-41f4-8521-0c7f53e7687f-istiod-ca-cert" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:38.672218 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.672190 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-envoy" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:38.672620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.672594 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-token" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:38.673149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.673128 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-kube-api-access-fdh5l" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "kube-api-access-fdh5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:38.673377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.673356 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-podinfo" pod "ac500e48-730e-41f4-8521-0c7f53e7687f" (UID: "ac500e48-730e-41f4-8521-0c7f53e7687f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:42:38.771812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.771777 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.771812 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.771813 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.772050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.771826 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac500e48-730e-41f4-8521-0c7f53e7687f-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.772050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.771840 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.772050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.771854 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.772050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.771867 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac500e48-730e-41f4-8521-0c7f53e7687f-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.772050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.771883 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdh5l\" (UniqueName: \"kubernetes.io/projected/ac500e48-730e-41f4-8521-0c7f53e7687f-kube-api-access-fdh5l\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:38.956385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.956296 4925 generic.go:356] "Generic (PLEG): container finished" podID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerID="920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2" exitCode=0 Nov 03 07:42:38.956385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.956362 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" Nov 03 07:42:38.956385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.956371 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" event={"ID":"ac500e48-730e-41f4-8521-0c7f53e7687f","Type":"ContainerDied","Data":"920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2"} Nov 03 07:42:38.956673 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.956439 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k" event={"ID":"ac500e48-730e-41f4-8521-0c7f53e7687f","Type":"ContainerDied","Data":"f9e2d8eb442c0bd2c850376142093d68fd769a600cf1e1c254e404ef1d71b65f"} Nov 03 07:42:38.956673 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.956471 4925 scope.go:122] "RemoveContainer" containerID="920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2" Nov 03 07:42:38.968783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.968762 4925 scope.go:122] "RemoveContainer" containerID="920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2" Nov 03 07:42:38.969182 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:42:38.969152 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2\": container with ID starting with 920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2 not found: ID does not exist" containerID="920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2" Nov 03 07:42:38.969291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:38.969194 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2"} err="failed to get container status \"920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2\": rpc error: code = NotFound desc = could not find container \"920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2\": container with ID starting with 920ee7d875846cf0eeab8305b650c46449667508b8ee7586d47305a071c22dc2 not found: ID does not exist" Nov 03 07:42:39.128713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:39.128677 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k"] Nov 03 07:42:39.138363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:39.138335 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k"] Nov 03 07:42:39.139083 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:42:39.139056 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:42:40.190475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:40.190439 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" path="/var/lib/kubelet/pods/ac500e48-730e-41f4-8521-0c7f53e7687f/volumes" Nov 03 07:42:40.749731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:40.749688 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd"] Nov 03 07:42:40.749922 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:40.749902 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" podUID="8145ea1e-95c7-47d8-a304-473757d53b74" containerName="maas-api" containerID="cri-o://6ee1a4db8237800154a5d10e3014d99559fa1da9d34772977c1038ee87c615d1" gracePeriod=30 Nov 03 07:42:40.912938 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:40.912903 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:41.008286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.008202 4925 generic.go:356] "Generic (PLEG): container finished" podID="8145ea1e-95c7-47d8-a304-473757d53b74" containerID="6ee1a4db8237800154a5d10e3014d99559fa1da9d34772977c1038ee87c615d1" exitCode=0 Nov 03 07:42:41.008286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.008260 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" event={"ID":"8145ea1e-95c7-47d8-a304-473757d53b74","Type":"ContainerDied","Data":"6ee1a4db8237800154a5d10e3014d99559fa1da9d34772977c1038ee87c615d1"} Nov 03 07:42:41.063736 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.063703 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:42:41.097335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.097297 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8145ea1e-95c7-47d8-a304-473757d53b74-kube-api-access-vdtgq\" (UniqueName: \"kubernetes.io/projected/8145ea1e-95c7-47d8-a304-473757d53b74-kube-api-access-vdtgq\") pod \"8145ea1e-95c7-47d8-a304-473757d53b74\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " Nov 03 07:42:41.097516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.097373 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-metrics-certs\") pod \"8145ea1e-95c7-47d8-a304-473757d53b74\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " Nov 03 07:42:41.097516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.097415 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls\") pod \"8145ea1e-95c7-47d8-a304-473757d53b74\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " Nov 03 07:42:41.097516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.097502 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8145ea1e-95c7-47d8-a304-473757d53b74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8145ea1e-95c7-47d8-a304-473757d53b74-openshift-service-ca\") pod \"8145ea1e-95c7-47d8-a304-473757d53b74\" (UID: \"8145ea1e-95c7-47d8-a304-473757d53b74\") " Nov 03 07:42:41.098249 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.098192 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8145ea1e-95c7-47d8-a304-473757d53b74-openshift-service-ca" pod "8145ea1e-95c7-47d8-a304-473757d53b74" (UID: "8145ea1e-95c7-47d8-a304-473757d53b74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:41.100270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.100206 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8145ea1e-95c7-47d8-a304-473757d53b74-kube-api-access-vdtgq" pod "8145ea1e-95c7-47d8-a304-473757d53b74" (UID: "8145ea1e-95c7-47d8-a304-473757d53b74"). InnerVolumeSpecName "kube-api-access-vdtgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:41.100473 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.100450 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls" pod "8145ea1e-95c7-47d8-a304-473757d53b74" (UID: "8145ea1e-95c7-47d8-a304-473757d53b74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:42:41.101416 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.101386 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-metrics-certs" pod "8145ea1e-95c7-47d8-a304-473757d53b74" (UID: "8145ea1e-95c7-47d8-a304-473757d53b74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:42:41.199294 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.199106 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdtgq\" (UniqueName: \"kubernetes.io/projected/8145ea1e-95c7-47d8-a304-473757d53b74-kube-api-access-vdtgq\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:41.199294 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.199141 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:41.199294 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.199156 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8145ea1e-95c7-47d8-a304-473757d53b74-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:41.199294 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:41.199169 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8145ea1e-95c7-47d8-a304-473757d53b74-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:42.016355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.016323 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" Nov 03 07:42:42.016355 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.016344 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd" event={"ID":"8145ea1e-95c7-47d8-a304-473757d53b74","Type":"ContainerDied","Data":"3dfc96bef6dbb441fe3f444cdd4c7b2dcaa21459200b229a6ab1c737b2438167"} Nov 03 07:42:42.016636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.016418 4925 scope.go:122] "RemoveContainer" containerID="6ee1a4db8237800154a5d10e3014d99559fa1da9d34772977c1038ee87c615d1" Nov 03 07:42:42.202589 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.202559 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd"] Nov 03 07:42:42.255548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.255506 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9697f7-58cbc8cbf6-klftd"] Nov 03 07:42:42.444618 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.444581 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:42.740291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.740204 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:42.950429 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:42.950363 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:43.415143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:43.415104 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:44.192358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:44.192324 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8145ea1e-95c7-47d8-a304-473757d53b74" path="/var/lib/kubelet/pods/8145ea1e-95c7-47d8-a304-473757d53b74/volumes" Nov 03 07:42:46.101775 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:46.101739 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:42:48.015554 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:48.015518 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:48.384505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:48.384469 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7"] Nov 03 07:42:48.384706 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:48.384668 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" containerID="cri-o://ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874" gracePeriod=30 Nov 03 07:42:50.122095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:50.121930 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:50.313766 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:50.313726 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:50.414282 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:50.414171 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh"] Nov 03 07:42:50.414456 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:50.414425 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" containerID="cri-o://83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc" gracePeriod=30 Nov 03 07:42:50.512874 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:50.512786 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:53.010887 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.010844 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:53.652073 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.651999 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:42:53.744663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744628 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-podinfo\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.744857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744678 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-envoy\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.744857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744726 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-data\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.744857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744772 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-credential-socket\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.744857 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744815 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-token\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.745201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744858 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-kube-api-access-j566t\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-kube-api-access-j566t\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.745201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744898 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-socket\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.745201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.744945 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-certs\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.745201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745079 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-data" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:53.745201 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745159 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istiod-ca-cert\") pod \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\" (UID: \"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f\") " Nov 03 07:42:53.745457 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745356 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-socket" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:53.745537 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745511 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-certs" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:53.745602 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745373 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-credential-socket" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:53.745791 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745765 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istiod-ca-cert" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:53.746042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745970 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.746042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.745999 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.746042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.746037 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.746415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.746051 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.746415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.746063 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.747310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.747281 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-podinfo" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:42:53.747884 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.747853 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-envoy" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:53.747986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.747958 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-token" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:53.748222 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.748189 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-kube-api-access-j566t" pod "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" (UID: "2b404f1b-84ed-4ce1-99e8-9ae38a66e17f"). InnerVolumeSpecName "kube-api-access-j566t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:53.847138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.847105 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.847138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.847133 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.847138 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.847143 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:53.847456 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:53.847151 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j566t\" (UniqueName: \"kubernetes.io/projected/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f-kube-api-access-j566t\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:54.097482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.097449 4925 generic.go:356] "Generic (PLEG): container finished" podID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerID="ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874" exitCode=0 Nov 03 07:42:54.097989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.097524 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" Nov 03 07:42:54.097989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.097535 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" event={"ID":"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f","Type":"ContainerDied","Data":"ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874"} Nov 03 07:42:54.097989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.097615 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7" event={"ID":"2b404f1b-84ed-4ce1-99e8-9ae38a66e17f","Type":"ContainerDied","Data":"a45e24b3cffdfb928da71cfd3c8735867c0873619f155db8fb3ae4b99da7ebfc"} Nov 03 07:42:54.097989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.097640 4925 scope.go:122] "RemoveContainer" containerID="ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874" Nov 03 07:42:54.109332 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.109306 4925 scope.go:122] "RemoveContainer" containerID="ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874" Nov 03 07:42:54.109617 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:42:54.109596 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874\": container with ID starting with ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874 not found: ID does not exist" containerID="ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874" Nov 03 07:42:54.109682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.109633 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874"} err="failed to get container status \"ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874\": rpc error: code = NotFound desc = could not find container \"ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874\": container with ID starting with ffdac378bbc9fd35138da831e07752bc658ef05c74625b3128e60a2d4e017874 not found: ID does not exist" Nov 03 07:42:54.270023 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.269988 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7"] Nov 03 07:42:54.320465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:54.320434 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7"] Nov 03 07:42:55.686405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.686352 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:42:55.768809 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.768776 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-envoy\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.768972 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.768825 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/953d85fc-693d-4793-a125-83b802aef073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953d85fc-693d-4793-a125-83b802aef073-istio-podinfo\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.768972 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.768884 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953d85fc-693d-4793-a125-83b802aef073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953d85fc-693d-4793-a125-83b802aef073-istiod-ca-cert\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.769092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769040 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-istio-token\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-istio-token\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.769144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769096 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-data\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.769199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769163 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-kube-api-access-zj7n5\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-kube-api-access-zj7n5\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.769258 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769219 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-socket\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.769318 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769273 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-credential-socket\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.769375 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769311 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-certs\") pod \"953d85fc-693d-4793-a125-83b802aef073\" (UID: \"953d85fc-693d-4793-a125-83b802aef073\") " Nov 03 07:42:55.769426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769402 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953d85fc-693d-4793-a125-83b802aef073-istiod-ca-cert" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:55.769486 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769404 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-data" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:55.769546 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769474 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-socket" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:55.769799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769775 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-credential-socket" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:55.769958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769797 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-certs" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:55.770106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769970 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953d85fc-693d-4793-a125-83b802aef073-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.770106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.769993 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.770106 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.770028 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.771398 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.771375 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-envoy" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:42:55.771504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.771490 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-istio-token" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:55.771931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.771910 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-kube-api-access-zj7n5" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "kube-api-access-zj7n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:55.771931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.771918 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/953d85fc-693d-4793-a125-83b802aef073-istio-podinfo" pod "953d85fc-693d-4793-a125-83b802aef073" (UID: "953d85fc-693d-4793-a125-83b802aef073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:42:55.871237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.871200 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7n5\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-kube-api-access-zj7n5\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.871237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.871229 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.871237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.871239 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.871471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.871248 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953d85fc-693d-4793-a125-83b802aef073-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.871471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.871257 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953d85fc-693d-4793-a125-83b802aef073-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:55.871471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:55.871265 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953d85fc-693d-4793-a125-83b802aef073-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:56.113557 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.113519 4925 generic.go:356] "Generic (PLEG): container finished" podID="953d85fc-693d-4793-a125-83b802aef073" containerID="83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc" exitCode=0 Nov 03 07:42:56.113753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.113565 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" event={"ID":"953d85fc-693d-4793-a125-83b802aef073","Type":"ContainerDied","Data":"83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc"} Nov 03 07:42:56.113753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.113596 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" Nov 03 07:42:56.113753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.113627 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh" event={"ID":"953d85fc-693d-4793-a125-83b802aef073","Type":"ContainerDied","Data":"e8c3c8f0e7364de687d7036e22b146419dcdc8c48166564c4e0e294f307b74b8"} Nov 03 07:42:56.113753 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.113650 4925 scope.go:122] "RemoveContainer" containerID="83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc" Nov 03 07:42:56.124569 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.124551 4925 scope.go:122] "RemoveContainer" containerID="83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc" Nov 03 07:42:56.124866 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:42:56.124848 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc\": container with ID starting with 83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc not found: ID does not exist" containerID="83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc" Nov 03 07:42:56.124915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.124874 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc"} err="failed to get container status \"83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc\": rpc error: code = NotFound desc = could not find container \"83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc\": container with ID starting with 83fcdcc059b3194ef1cb100ab1ad2a2c47043d8c20568fe09360321350b79ddc not found: ID does not exist" Nov 03 07:42:56.190773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.190743 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" path="/var/lib/kubelet/pods/2b404f1b-84ed-4ce1-99e8-9ae38a66e17f/volumes" Nov 03 07:42:56.287519 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.287485 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh"] Nov 03 07:42:56.292605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:56.292578 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh"] Nov 03 07:42:58.022673 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:58.022638 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:42:58.193869 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:42:58.193832 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953d85fc-693d-4793-a125-83b802aef073" path="/var/lib/kubelet/pods/953d85fc-693d-4793-a125-83b802aef073/volumes" Nov 03 07:43:03.068456 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.068417 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:03.791854 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.791815 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb"] Nov 03 07:43:03.792291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792279 4925 cpu_manager.go:414] "containerMap: removing container" podUID="8145ea1e-95c7-47d8-a304-473757d53b74" containerName="maas-api" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792292 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8145ea1e-95c7-47d8-a304-473757d53b74" containerName="maas-api" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792305 4925 cpu_manager.go:414] "containerMap: removing container" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792310 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792318 4925 cpu_manager.go:414] "containerMap: removing container" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792322 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792337 4925 cpu_manager.go:414] "containerMap: removing container" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792342 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" Nov 03 07:43:03.792352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792349 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" containerName="payload-pre-processing" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792354 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" containerName="payload-pre-processing" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792365 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b61eccd2-6a03-4dd9-829f-414965b61fa1" containerName="maas-api" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792369 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61eccd2-6a03-4dd9-829f-414965b61fa1" containerName="maas-api" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792382 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792387 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792394 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792398 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792405 4925 cpu_manager.go:414] "containerMap: removing container" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792410 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792419 4925 cpu_manager.go:414] "containerMap: removing container" podUID="be964d3c-36c1-43fd-b8da-b5483f6819ac" containerName="payload-processing" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792423 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be964d3c-36c1-43fd-b8da-b5483f6819ac" containerName="payload-processing" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792431 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792435 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792509 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="be964d3c-36c1-43fd-b8da-b5483f6819ac" containerName="payload-processing" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792520 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a95df8-8830-4852-af2b-ecf9b484bf3c" containerName="main" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792528 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8145ea1e-95c7-47d8-a304-473757d53b74" containerName="maas-api" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792534 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef01dc3-a11f-461c-8d55-78763b38f002" containerName="main" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792540 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5a47fd-67ac-479c-8064-b067ea059c6f" containerName="main" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792548 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac500e48-730e-41f4-8521-0c7f53e7687f" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792553 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ae1ea2-c7c3-4427-af01-f73d09e48cd2" containerName="payload-pre-processing" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792559 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61eccd2-6a03-4dd9-829f-414965b61fa1" containerName="maas-api" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792565 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a589ad64-54f2-4ae8-8844-351469ef11f2" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792570 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b404f1b-84ed-4ce1-99e8-9ae38a66e17f" containerName="istio-proxy" Nov 03 07:43:03.792623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.792576 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="953d85fc-693d-4793-a125-83b802aef073" containerName="istio-proxy" Nov 03 07:43:03.796798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.796783 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.861146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.861113 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjnq\" (UniqueName: \"kubernetes.io/projected/ac12005a-d91c-404f-ab8c-5c94f723b2fc-kube-api-access-7cjnq\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.861339 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.861160 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-dshm\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.861411 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.861331 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tmp-dir\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.861411 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.861404 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tls-certs\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.861505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.861477 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-model-cache\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.861563 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.861547 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-home\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.902353 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.902327 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50ae96\"/\"default-dockercfg-tktsv\"" Nov 03 07:43:03.902529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.902355 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50ae96\"/\"openshift-service-ca.crt\"" Nov 03 07:43:03.902529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.902408 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50ae96\"/\"test-model-50ae96-kserve-self-signed-certs\"" Nov 03 07:43:03.902529 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.902477 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50ae96\"/\"kube-root-ca.crt\"" Nov 03 07:43:03.905577 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.905542 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb"] Nov 03 07:43:03.952025 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.951962 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:03.962855 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.962814 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjnq\" (UniqueName: \"kubernetes.io/projected/ac12005a-d91c-404f-ab8c-5c94f723b2fc-kube-api-access-7cjnq\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.962863 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-dshm\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.962912 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tmp-dir\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.962947 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tls-certs\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.962978 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-model-cache\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.963064 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-home\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963397 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.963373 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tmp-dir\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.963477 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-model-cache\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.963638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.963532 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-home\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.965999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.965973 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-dshm\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:03.966580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:03.966543 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tls-certs\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:04.030323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:04.030293 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjnq\" (UniqueName: \"kubernetes.io/projected/ac12005a-d91c-404f-ab8c-5c94f723b2fc-kube-api-access-7cjnq\") pod \"test-model-50ae96-kserve-5968d5fcdc-rvlrb\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:04.108701 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:04.108672 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:04.489385 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:04.489359 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb"] Nov 03 07:43:04.490543 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:43:04.490506 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac12005a_d91c_404f_ab8c_5c94f723b2fc.slice/crio-88a86e2a55dce804d21e0816bb32f88b09e300a94962da335669842e1ae24671 WatchSource:0}: Error finding container 88a86e2a55dce804d21e0816bb32f88b09e300a94962da335669842e1ae24671: Status 404 returned error can't find the container with id 88a86e2a55dce804d21e0816bb32f88b09e300a94962da335669842e1ae24671 Nov 03 07:43:05.175615 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:05.175575 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" event={"ID":"ac12005a-d91c-404f-ab8c-5c94f723b2fc","Type":"ContainerStarted","Data":"263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d"} Nov 03 07:43:05.176003 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:05.175625 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" event={"ID":"ac12005a-d91c-404f-ab8c-5c94f723b2fc","Type":"ContainerStarted","Data":"88a86e2a55dce804d21e0816bb32f88b09e300a94962da335669842e1ae24671"} Nov 03 07:43:05.297195 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:05.297144 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" podStartSLOduration=2.29712995 podStartE2EDuration="2.29712995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:43:05.296213361 +0000 UTC m=+2645.229742017" watchObservedRunningTime="2026-11-03 07:43:05.29712995 +0000 UTC m=+2645.230658607" Nov 03 07:43:05.795221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:05.795185 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:06.109299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:06.109268 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:08.008760 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:08.008717 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:09.138780 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:43:09.138745 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:43:12.956782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:12.956558 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:14.109646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:14.109604 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:14.122307 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:14.122279 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:14.241626 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:14.241593 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:43:17.292778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.292736 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf"] Nov 03 07:43:17.296919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.296895 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.330585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.330554 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkgs\" (UniqueName: \"kubernetes.io/projected/d344362f-1e23-4c7d-9829-4a989600135a-kube-api-access-5kkgs\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.330726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.330603 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-tmp-dir\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.330783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.330746 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-model-cache\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.330783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.330777 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-dshm\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.330875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.330800 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-home\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.330919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.330896 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d344362f-1e23-4c7d-9829-4a989600135a-tls-certs\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.357439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.357405 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf"] Nov 03 07:43:17.399287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.399263 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eac348\"/\"openshift-service-ca.crt\"" Nov 03 07:43:17.403830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.403811 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eac348\"/\"kube-root-ca.crt\"" Nov 03 07:43:17.403952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.403826 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eac348\"/\"default-dockercfg-524gq\"" Nov 03 07:43:17.403952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.403893 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eac348\"/\"test-model-eac348-kserve-self-signed-certs\"" Nov 03 07:43:17.432102 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432069 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-model-cache\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432126 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-dshm\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432164 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-home\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432221 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d344362f-1e23-4c7d-9829-4a989600135a-tls-certs\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432291 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkgs\" (UniqueName: \"kubernetes.io/projected/d344362f-1e23-4c7d-9829-4a989600135a-kube-api-access-5kkgs\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432326 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-tmp-dir\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432949 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432800 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-tmp-dir\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432949 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432871 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-home\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.432949 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.432884 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-model-cache\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.435776 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.435748 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-dshm\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.436033 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.435960 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:17.436290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.436269 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d344362f-1e23-4c7d-9829-4a989600135a-tls-certs\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.496999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.496957 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkgs\" (UniqueName: \"kubernetes.io/projected/d344362f-1e23-4c7d-9829-4a989600135a-kube-api-access-5kkgs\") pod \"test-model-eac348-kserve-57b94d7759-6jzdf\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.610152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.610111 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:17.948848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:17.948744 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:18.007295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:18.007261 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf"] Nov 03 07:43:18.256389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:18.256302 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" event={"ID":"d344362f-1e23-4c7d-9829-4a989600135a","Type":"ContainerStarted","Data":"6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1"} Nov 03 07:43:18.256389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:18.256340 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" event={"ID":"d344362f-1e23-4c7d-9829-4a989600135a","Type":"ContainerStarted","Data":"036a5f2facedaa27c4098d679ee351f32133d3e6ef3812bca373209a4a4cecbc"} Nov 03 07:43:18.379268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:18.379212 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" podStartSLOduration=1.37919829 podStartE2EDuration="1.37919829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:43:18.377253941 +0000 UTC m=+2658.310782597" watchObservedRunningTime="2026-11-03 07:43:18.37919829 +0000 UTC m=+2658.312726946" Nov 03 07:43:18.610718 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:18.610682 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:23.034459 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:23.034420 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:27.610550 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:27.610506 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:27.623236 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:27.623203 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:28.176142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:28.176100 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:28.331797 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:28.331765 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:43:29.736293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:29.736252 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:43:33.317752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:33.317715 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:37.585580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:37.585543 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96"] Nov 03 07:43:37.586146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:37.585782 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" podUID="972f9a40-0986-497e-8720-5876a4dbfd5c" containerName="maas-api" containerID="cri-o://0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5" gracePeriod=30 Nov 03 07:43:37.744866 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:37.744828 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:37.891578 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:37.891545 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:43:38.065032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.064982 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972f9a40-0986-497e-8720-5876a4dbfd5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f9a40-0986-497e-8720-5876a4dbfd5c-openshift-service-ca\") pod \"972f9a40-0986-497e-8720-5876a4dbfd5c\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " Nov 03 07:43:38.065226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.065070 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls\") pod \"972f9a40-0986-497e-8720-5876a4dbfd5c\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " Nov 03 07:43:38.065226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.065132 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972f9a40-0986-497e-8720-5876a4dbfd5c-kube-api-access-k9wqr\" (UniqueName: \"kubernetes.io/projected/972f9a40-0986-497e-8720-5876a4dbfd5c-kube-api-access-k9wqr\") pod \"972f9a40-0986-497e-8720-5876a4dbfd5c\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " Nov 03 07:43:38.065368 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.065329 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-metrics-certs\") pod \"972f9a40-0986-497e-8720-5876a4dbfd5c\" (UID: \"972f9a40-0986-497e-8720-5876a4dbfd5c\") " Nov 03 07:43:38.066796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.066759 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972f9a40-0986-497e-8720-5876a4dbfd5c-openshift-service-ca" pod "972f9a40-0986-497e-8720-5876a4dbfd5c" (UID: "972f9a40-0986-497e-8720-5876a4dbfd5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:43:38.069516 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.069474 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-metrics-certs" pod "972f9a40-0986-497e-8720-5876a4dbfd5c" (UID: "972f9a40-0986-497e-8720-5876a4dbfd5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:43:38.071641 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.071609 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls" pod "972f9a40-0986-497e-8720-5876a4dbfd5c" (UID: "972f9a40-0986-497e-8720-5876a4dbfd5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:43:38.073482 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.073448 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972f9a40-0986-497e-8720-5876a4dbfd5c-kube-api-access-k9wqr" pod "972f9a40-0986-497e-8720-5876a4dbfd5c" (UID: "972f9a40-0986-497e-8720-5876a4dbfd5c"). InnerVolumeSpecName "kube-api-access-k9wqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:43:38.166582 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.166497 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:38.166582 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.166530 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f9a40-0986-497e-8720-5876a4dbfd5c-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:38.166582 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.166541 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f9a40-0986-497e-8720-5876a4dbfd5c-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:38.166582 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.166551 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9wqr\" (UniqueName: \"kubernetes.io/projected/972f9a40-0986-497e-8720-5876a4dbfd5c-kube-api-access-k9wqr\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:38.386286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.386251 4925 generic.go:356] "Generic (PLEG): container finished" podID="972f9a40-0986-497e-8720-5876a4dbfd5c" containerID="0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5" exitCode=0 Nov 03 07:43:38.386503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.386301 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" Nov 03 07:43:38.386503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.386347 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" event={"ID":"972f9a40-0986-497e-8720-5876a4dbfd5c","Type":"ContainerDied","Data":"0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5"} Nov 03 07:43:38.386503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.386416 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96" event={"ID":"972f9a40-0986-497e-8720-5876a4dbfd5c","Type":"ContainerDied","Data":"9dd24310804becf5c392f92f4cc3914bb0dbe344e7e58fe2bb94fdd5c4295d5d"} Nov 03 07:43:38.386503 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.386447 4925 scope.go:122] "RemoveContainer" containerID="0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5" Nov 03 07:43:38.398057 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.398033 4925 scope.go:122] "RemoveContainer" containerID="0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5" Nov 03 07:43:38.398357 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:43:38.398326 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5\": container with ID starting with 0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5 not found: ID does not exist" containerID="0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5" Nov 03 07:43:38.398435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.398364 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5"} err="failed to get container status \"0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5\": rpc error: code = NotFound desc = could not find container \"0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5\": container with ID starting with 0b9bbe0c52e2d55fbf82ac3d216b5f1c6abf131f19ff848c51f6707127d2c1f5 not found: ID does not exist" Nov 03 07:43:38.598199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.577182 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96"] Nov 03 07:43:38.600888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.600845 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a8983d-f8c7cc7fd-dvz96"] Nov 03 07:43:38.969378 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:38.969279 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:39.140217 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:43:39.140177 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:43:39.360569 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:39.360524 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:39.642672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:39.642592 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:40.194626 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:40.194592 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972f9a40-0986-497e-8720-5876a4dbfd5c" path="/var/lib/kubelet/pods/972f9a40-0986-497e-8720-5876a4dbfd5c/volumes" Nov 03 07:43:40.342066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:40.342005 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:40.652388 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:40.652348 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:40.920976 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:40.920892 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:41.121123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:41.121077 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:41.658890 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:41.658852 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:46.355166 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:46.355125 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:48.161029 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:48.160975 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:48.460555 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:48.460451 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:48.667903 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:48.667871 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:49.388763 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:49.388730 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:49.619638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:49.619599 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:49.823735 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:49.823696 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:51.861379 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:51.861341 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:43:57.099445 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:43:57.099393 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:01.358123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:01.358086 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:01.576377 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:01.576337 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:01.847830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:01.847802 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:44:01.851948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:01.851920 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:44:01.853634 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:01.853613 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:44:01.857335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:01.857313 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:44:02.158254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:02.158225 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:03.907499 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:03.907459 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:04.125027 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:04.124984 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:06.180940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:06.180881 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:06.446896 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:06.446809 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:07.256492 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:07.256451 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:09.138004 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:44:09.137966 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:44:12.254380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:12.254341 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:17.237807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:17.237768 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:22.234463 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:22.234424 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:25.160451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:25.160411 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:25.403197 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:25.403159 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:27.093616 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:27.093572 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:44:27.313063 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:27.313000 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:32.324126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:32.324088 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:37.430653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:37.430611 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:39.138533 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:44:39.138490 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:44:42.382595 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:42.382559 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:47.474671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:47.474632 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:52.523348 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:52.523314 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:44:56.892952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:56.892914 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:44:57.459226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:44:57.459183 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:00.203939 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.203903 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj"] Nov 03 07:45:00.204405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.204392 4925 cpu_manager.go:414] "containerMap: removing container" podUID="972f9a40-0986-497e-8720-5876a4dbfd5c" containerName="maas-api" Nov 03 07:45:00.204450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.204407 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972f9a40-0986-497e-8720-5876a4dbfd5c" containerName="maas-api" Nov 03 07:45:00.204514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.204505 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="972f9a40-0986-497e-8720-5876a4dbfd5c" containerName="maas-api" Nov 03 07:45:00.207591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.207575 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.233321 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.233296 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w"] Nov 03 07:45:00.236845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.236825 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7"] Nov 03 07:45:00.236948 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.236931 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.240508 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.240492 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.256247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.256223 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb"] Nov 03 07:45:00.269622 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.269592 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.293918 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.293877 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788q9\" (UniqueName: \"kubernetes.io/projected/6687c623-2897-4891-aefe-b9d285910a84-kube-api-access-788q9\") pod \"maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.294119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.293981 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e52396-ef05-4008-919f-1610ce4a4aab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.294119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.294041 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnsg\" (UniqueName: \"kubernetes.io/projected/38515a4d-17c9-45dc-995a-dea48bd883dc-kube-api-access-srnsg\") pod \"maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.294263 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.294155 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38515a4d-17c9-45dc-995a-dea48bd883dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.294263 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.294184 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxjb\" (UniqueName: \"kubernetes.io/projected/39e52396-ef05-4008-919f-1610ce4a4aab-kube-api-access-7wxjb\") pod \"maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.294263 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.294201 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7756b04-4fca-44d7-af02-0bef47d8ae3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.294263 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.294218 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6687c623-2897-4891-aefe-b9d285910a84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.294487 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.294385 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftxs\" (UniqueName: \"kubernetes.io/projected/e7756b04-4fca-44d7-af02-0bef47d8ae3b-kube-api-access-tftxs\") pod \"maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.298447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.298420 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj"] Nov 03 07:45:00.347111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.347080 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w"] Nov 03 07:45:00.352464 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.352426 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb"] Nov 03 07:45:00.355917 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.355878 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:45:00.356130 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.356077 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7"] Nov 03 07:45:00.395617 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395581 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38515a4d-17c9-45dc-995a-dea48bd883dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.395797 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395623 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxjb\" (UniqueName: \"kubernetes.io/projected/39e52396-ef05-4008-919f-1610ce4a4aab-kube-api-access-7wxjb\") pod \"maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.395797 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395653 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7756b04-4fca-44d7-af02-0bef47d8ae3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.395797 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395677 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6687c623-2897-4891-aefe-b9d285910a84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.395797 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395751 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftxs\" (UniqueName: \"kubernetes.io/projected/e7756b04-4fca-44d7-af02-0bef47d8ae3b-kube-api-access-tftxs\") pod \"maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.396088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395858 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788q9\" (UniqueName: \"kubernetes.io/projected/6687c623-2897-4891-aefe-b9d285910a84-kube-api-access-788q9\") pod \"maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.396088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395909 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e52396-ef05-4008-919f-1610ce4a4aab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.396088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.395945 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnsg\" (UniqueName: \"kubernetes.io/projected/38515a4d-17c9-45dc-995a-dea48bd883dc-kube-api-access-srnsg\") pod \"maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.396486 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.396458 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7756b04-4fca-44d7-af02-0bef47d8ae3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.396727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.396473 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38515a4d-17c9-45dc-995a-dea48bd883dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.396843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.396809 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6687c623-2897-4891-aefe-b9d285910a84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.397111 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.397088 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e52396-ef05-4008-919f-1610ce4a4aab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.460782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.460701 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788q9\" (UniqueName: \"kubernetes.io/projected/6687c623-2897-4891-aefe-b9d285910a84-kube-api-access-788q9\") pod \"maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.460945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.460789 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxjb\" (UniqueName: \"kubernetes.io/projected/39e52396-ef05-4008-919f-1610ce4a4aab-kube-api-access-7wxjb\") pod \"maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.460945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.460857 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftxs\" (UniqueName: \"kubernetes.io/projected/e7756b04-4fca-44d7-af02-0bef47d8ae3b-kube-api-access-tftxs\") pod \"maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.460945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.460914 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnsg\" (UniqueName: \"kubernetes.io/projected/38515a4d-17c9-45dc-995a-dea48bd883dc-kube-api-access-srnsg\") pod \"maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.573582 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.573548 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:45:00.578716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.578690 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:00.582031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.581981 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:00.582223 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.582211 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:00.584422 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.584386 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:00.987782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:00.987747 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7"] Nov 03 07:45:00.988179 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:45:00.988155 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38515a4d_17c9_45dc_995a_dea48bd883dc.slice/crio-3d3c6b50e6eb81505f154af10b2017ebe8fde38fb4a87e1d68b4adb32007177d WatchSource:0}: Error finding container 3d3c6b50e6eb81505f154af10b2017ebe8fde38fb4a87e1d68b4adb32007177d: Status 404 returned error can't find the container with id 3d3c6b50e6eb81505f154af10b2017ebe8fde38fb4a87e1d68b4adb32007177d Nov 03 07:45:01.032943 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.032910 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj"] Nov 03 07:45:01.034220 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:45:01.034178 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e52396_ef05_4008_919f_1610ce4a4aab.slice/crio-0caa7d89713f3f036b7b1f4a31242e47b5ac6dc10683283f50a7d1234efea766 WatchSource:0}: Error finding container 0caa7d89713f3f036b7b1f4a31242e47b5ac6dc10683283f50a7d1234efea766: Status 404 returned error can't find the container with id 0caa7d89713f3f036b7b1f4a31242e47b5ac6dc10683283f50a7d1234efea766 Nov 03 07:45:01.036499 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:45:01.036476 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7756b04_4fca_44d7_af02_0bef47d8ae3b.slice/crio-4f33eabea42007b68861a3ead2541deca8cd85d54e7078c6527ee7635b852c0e WatchSource:0}: Error finding container 4f33eabea42007b68861a3ead2541deca8cd85d54e7078c6527ee7635b852c0e: Status 404 returned error can't find the container with id 4f33eabea42007b68861a3ead2541deca8cd85d54e7078c6527ee7635b852c0e Nov 03 07:45:01.036906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.036878 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb"] Nov 03 07:45:01.039469 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:45:01.039441 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6687c623_2897_4891_aefe_b9d285910a84.slice/crio-601b2fedd2fe520841b0a6c089afd3a9561bd15f69930ff3af85e7821bdd8208 WatchSource:0}: Error finding container 601b2fedd2fe520841b0a6c089afd3a9561bd15f69930ff3af85e7821bdd8208: Status 404 returned error can't find the container with id 601b2fedd2fe520841b0a6c089afd3a9561bd15f69930ff3af85e7821bdd8208 Nov 03 07:45:01.040297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.040256 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w"] Nov 03 07:45:01.942858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.942815 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" event={"ID":"e7756b04-4fca-44d7-af02-0bef47d8ae3b","Type":"ContainerStarted","Data":"bea13d16e43eed2112f7e074f7dfd4a722347e6a0d25de0682b087db857077a4"} Nov 03 07:45:01.943358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.942881 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" event={"ID":"e7756b04-4fca-44d7-af02-0bef47d8ae3b","Type":"ContainerStarted","Data":"4f33eabea42007b68861a3ead2541deca8cd85d54e7078c6527ee7635b852c0e"} Nov 03 07:45:01.944521 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.944475 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" event={"ID":"39e52396-ef05-4008-919f-1610ce4a4aab","Type":"ContainerStarted","Data":"71c8d00af3eed73e1fcc4b68200c43722458bdaab591c090fde98359138e3914"} Nov 03 07:45:01.944678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.944529 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" event={"ID":"39e52396-ef05-4008-919f-1610ce4a4aab","Type":"ContainerStarted","Data":"0caa7d89713f3f036b7b1f4a31242e47b5ac6dc10683283f50a7d1234efea766"} Nov 03 07:45:01.947247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.947219 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerStarted","Data":"3290ec3d7317083b30f7f76201316885e06fb17e778dac1d79b5c258b77fe47a"} Nov 03 07:45:01.947659 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.947270 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerStarted","Data":"601b2fedd2fe520841b0a6c089afd3a9561bd15f69930ff3af85e7821bdd8208"} Nov 03 07:45:01.949335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.949304 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" event={"ID":"38515a4d-17c9-45dc-995a-dea48bd883dc","Type":"ContainerStarted","Data":"82774795001e0be0b27be4b9b377d600b8ee9b5418b958c080c66805d5668868"} Nov 03 07:45:01.949454 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:01.949362 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" event={"ID":"38515a4d-17c9-45dc-995a-dea48bd883dc","Type":"ContainerStarted","Data":"3d3c6b50e6eb81505f154af10b2017ebe8fde38fb4a87e1d68b4adb32007177d"} Nov 03 07:45:02.064001 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.063958 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" podStartSLOduration=2.063943927 podStartE2EDuration="2.063943927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:02.063746655 +0000 UTC m=+2761.997275310" watchObservedRunningTime="2026-11-03 07:45:02.063943927 +0000 UTC m=+2761.997472583" Nov 03 07:45:02.188055 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.187978 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" podStartSLOduration=2.18796078 podStartE2EDuration="2.18796078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:02.18555287 +0000 UTC m=+2762.119081526" watchObservedRunningTime="2026-11-03 07:45:02.18796078 +0000 UTC m=+2762.121489440" Nov 03 07:45:02.307663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.307611 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" podStartSLOduration=2.307597302 podStartE2EDuration="2.307597302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:02.305231063 +0000 UTC m=+2762.238759718" watchObservedRunningTime="2026-11-03 07:45:02.307597302 +0000 UTC m=+2762.241125974" Nov 03 07:45:02.426885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.426828 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" podStartSLOduration=2.4268126199999998 podStartE2EDuration="2.42681262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:45:02.424025014 +0000 UTC m=+2762.357553663" watchObservedRunningTime="2026-11-03 07:45:02.42681262 +0000 UTC m=+2762.360341276" Nov 03 07:45:02.568240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.568207 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:02.957134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.957051 4925 generic.go:356] "Generic (PLEG): container finished" podID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerID="bea13d16e43eed2112f7e074f7dfd4a722347e6a0d25de0682b087db857077a4" exitCode=60 Nov 03 07:45:02.957134 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.957105 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" event={"ID":"e7756b04-4fca-44d7-af02-0bef47d8ae3b","Type":"ContainerDied","Data":"bea13d16e43eed2112f7e074f7dfd4a722347e6a0d25de0682b087db857077a4"} Nov 03 07:45:02.957596 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.957438 4925 scope.go:122] "RemoveContainer" containerID="bea13d16e43eed2112f7e074f7dfd4a722347e6a0d25de0682b087db857077a4" Nov 03 07:45:02.958971 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.958945 4925 generic.go:356] "Generic (PLEG): container finished" podID="39e52396-ef05-4008-919f-1610ce4a4aab" containerID="71c8d00af3eed73e1fcc4b68200c43722458bdaab591c090fde98359138e3914" exitCode=60 Nov 03 07:45:02.959129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.959058 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" event={"ID":"39e52396-ef05-4008-919f-1610ce4a4aab","Type":"ContainerDied","Data":"71c8d00af3eed73e1fcc4b68200c43722458bdaab591c090fde98359138e3914"} Nov 03 07:45:02.959286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.959269 4925 scope.go:122] "RemoveContainer" containerID="71c8d00af3eed73e1fcc4b68200c43722458bdaab591c090fde98359138e3914" Nov 03 07:45:02.961288 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.961268 4925 generic.go:356] "Generic (PLEG): container finished" podID="6687c623-2897-4891-aefe-b9d285910a84" containerID="3290ec3d7317083b30f7f76201316885e06fb17e778dac1d79b5c258b77fe47a" exitCode=60 Nov 03 07:45:02.961382 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.961359 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerDied","Data":"3290ec3d7317083b30f7f76201316885e06fb17e778dac1d79b5c258b77fe47a"} Nov 03 07:45:02.961520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.961506 4925 scope.go:122] "RemoveContainer" containerID="3290ec3d7317083b30f7f76201316885e06fb17e778dac1d79b5c258b77fe47a" Nov 03 07:45:02.962969 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.962947 4925 generic.go:356] "Generic (PLEG): container finished" podID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerID="82774795001e0be0b27be4b9b377d600b8ee9b5418b958c080c66805d5668868" exitCode=60 Nov 03 07:45:02.963054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.962990 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" event={"ID":"38515a4d-17c9-45dc-995a-dea48bd883dc","Type":"ContainerDied","Data":"82774795001e0be0b27be4b9b377d600b8ee9b5418b958c080c66805d5668868"} Nov 03 07:45:02.963290 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:02.963274 4925 scope.go:122] "RemoveContainer" containerID="82774795001e0be0b27be4b9b377d600b8ee9b5418b958c080c66805d5668868" Nov 03 07:45:03.973062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.973004 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerStarted","Data":"75660a95c727a303dea5580a9da815dbf0fd885c1fd3bb848f105b3f83feb990"} Nov 03 07:45:03.975245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.975223 4925 generic.go:356] "Generic (PLEG): container finished" podID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerID="eed68d57581ef4871a75832f0902cb16c79947779a88497bcd32c26e1c5f3cbc" exitCode=60 Nov 03 07:45:03.975367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.975302 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" event={"ID":"38515a4d-17c9-45dc-995a-dea48bd883dc","Type":"ContainerDied","Data":"eed68d57581ef4871a75832f0902cb16c79947779a88497bcd32c26e1c5f3cbc"} Nov 03 07:45:03.975367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.975360 4925 scope.go:122] "RemoveContainer" containerID="82774795001e0be0b27be4b9b377d600b8ee9b5418b958c080c66805d5668868" Nov 03 07:45:03.975575 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.975557 4925 scope.go:122] "RemoveContainer" containerID="eed68d57581ef4871a75832f0902cb16c79947779a88497bcd32c26e1c5f3cbc" Nov 03 07:45:03.975867 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:03.975846 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7_odh-ai-gateway-infra(38515a4d-17c9-45dc-995a-dea48bd883dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" Nov 03 07:45:03.977534 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.977512 4925 generic.go:356] "Generic (PLEG): container finished" podID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerID="015f5453ef84767843dae9ed5d08fb34f9a482642073dbf1eb980b893e936644" exitCode=60 Nov 03 07:45:03.977614 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.977591 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" event={"ID":"e7756b04-4fca-44d7-af02-0bef47d8ae3b","Type":"ContainerDied","Data":"015f5453ef84767843dae9ed5d08fb34f9a482642073dbf1eb980b893e936644"} Nov 03 07:45:03.977788 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.977773 4925 scope.go:122] "RemoveContainer" containerID="015f5453ef84767843dae9ed5d08fb34f9a482642073dbf1eb980b893e936644" Nov 03 07:45:03.978027 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:03.977989 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w_odh-ai-gateway-infra(e7756b04-4fca-44d7-af02-0bef47d8ae3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" Nov 03 07:45:03.979495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:03.979473 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" event={"ID":"39e52396-ef05-4008-919f-1610ce4a4aab","Type":"ContainerStarted","Data":"1f726af94d53fc15f246da02a919594a0299d056f0bbebe7b2a9fa636a25d7c3"} Nov 03 07:45:04.018435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.018416 4925 scope.go:122] "RemoveContainer" containerID="bea13d16e43eed2112f7e074f7dfd4a722347e6a0d25de0682b087db857077a4" Nov 03 07:45:04.988092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.988063 4925 scope.go:122] "RemoveContainer" containerID="015f5453ef84767843dae9ed5d08fb34f9a482642073dbf1eb980b893e936644" Nov 03 07:45:04.988593 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:04.988370 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w_odh-ai-gateway-infra(e7756b04-4fca-44d7-af02-0bef47d8ae3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" Nov 03 07:45:04.989802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.989769 4925 generic.go:356] "Generic (PLEG): container finished" podID="39e52396-ef05-4008-919f-1610ce4a4aab" containerID="1f726af94d53fc15f246da02a919594a0299d056f0bbebe7b2a9fa636a25d7c3" exitCode=60 Nov 03 07:45:04.989916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.989851 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" event={"ID":"39e52396-ef05-4008-919f-1610ce4a4aab","Type":"ContainerDied","Data":"1f726af94d53fc15f246da02a919594a0299d056f0bbebe7b2a9fa636a25d7c3"} Nov 03 07:45:04.989916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.989906 4925 scope.go:122] "RemoveContainer" containerID="71c8d00af3eed73e1fcc4b68200c43722458bdaab591c090fde98359138e3914" Nov 03 07:45:04.990129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.990110 4925 scope.go:122] "RemoveContainer" containerID="1f726af94d53fc15f246da02a919594a0299d056f0bbebe7b2a9fa636a25d7c3" Nov 03 07:45:04.990398 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:04.990375 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj_odh-ai-gateway-infra(39e52396-ef05-4008-919f-1610ce4a4aab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" Nov 03 07:45:04.992161 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.992136 4925 generic.go:356] "Generic (PLEG): container finished" podID="6687c623-2897-4891-aefe-b9d285910a84" containerID="75660a95c727a303dea5580a9da815dbf0fd885c1fd3bb848f105b3f83feb990" exitCode=60 Nov 03 07:45:04.992275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.992223 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerDied","Data":"75660a95c727a303dea5580a9da815dbf0fd885c1fd3bb848f105b3f83feb990"} Nov 03 07:45:04.992414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.992401 4925 scope.go:122] "RemoveContainer" containerID="75660a95c727a303dea5580a9da815dbf0fd885c1fd3bb848f105b3f83feb990" Nov 03 07:45:04.992684 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:04.992662 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb_odh-ai-gateway-infra(6687c623-2897-4891-aefe-b9d285910a84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" podUID="6687c623-2897-4891-aefe-b9d285910a84" Nov 03 07:45:04.993968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:04.993946 4925 scope.go:122] "RemoveContainer" containerID="eed68d57581ef4871a75832f0902cb16c79947779a88497bcd32c26e1c5f3cbc" Nov 03 07:45:04.994192 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:04.994173 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7_odh-ai-gateway-infra(38515a4d-17c9-45dc-995a-dea48bd883dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" Nov 03 07:45:05.002517 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:05.002494 4925 scope.go:122] "RemoveContainer" containerID="3290ec3d7317083b30f7f76201316885e06fb17e778dac1d79b5c258b77fe47a" Nov 03 07:45:06.001928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:06.001895 4925 scope.go:122] "RemoveContainer" containerID="75660a95c727a303dea5580a9da815dbf0fd885c1fd3bb848f105b3f83feb990" Nov 03 07:45:06.002445 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:06.002241 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb_odh-ai-gateway-infra(6687c623-2897-4891-aefe-b9d285910a84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" podUID="6687c623-2897-4891-aefe-b9d285910a84" Nov 03 07:45:06.003715 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:06.003688 4925 scope.go:122] "RemoveContainer" containerID="1f726af94d53fc15f246da02a919594a0299d056f0bbebe7b2a9fa636a25d7c3" Nov 03 07:45:06.003942 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:06.003920 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj_odh-ai-gateway-infra(39e52396-ef05-4008-919f-1610ce4a4aab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" Nov 03 07:45:06.502966 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:06.502926 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:07.589430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:07.589387 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:09.138677 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:09.138640 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:45:12.528762 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:12.528720 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:15.184220 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:15.184183 4925 scope.go:122] "RemoveContainer" containerID="015f5453ef84767843dae9ed5d08fb34f9a482642073dbf1eb980b893e936644" Nov 03 07:45:16.071199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:16.071167 4925 generic.go:356] "Generic (PLEG): container finished" podID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerID="bfb8f89c811da0989996d68a7fc63b8385b62333fc06a669452915073ad968ba" exitCode=60 Nov 03 07:45:16.071394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:16.071247 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" event={"ID":"e7756b04-4fca-44d7-af02-0bef47d8ae3b","Type":"ContainerDied","Data":"bfb8f89c811da0989996d68a7fc63b8385b62333fc06a669452915073ad968ba"} Nov 03 07:45:16.071394 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:16.071317 4925 scope.go:122] "RemoveContainer" containerID="015f5453ef84767843dae9ed5d08fb34f9a482642073dbf1eb980b893e936644" Nov 03 07:45:16.071730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:16.071705 4925 scope.go:122] "RemoveContainer" containerID="bfb8f89c811da0989996d68a7fc63b8385b62333fc06a669452915073ad968ba" Nov 03 07:45:16.072058 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:16.072031 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w_odh-ai-gateway-infra(e7756b04-4fca-44d7-af02-0bef47d8ae3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" Nov 03 07:45:16.184296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:16.184251 4925 scope.go:122] "RemoveContainer" containerID="eed68d57581ef4871a75832f0902cb16c79947779a88497bcd32c26e1c5f3cbc" Nov 03 07:45:17.080720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.080686 4925 generic.go:356] "Generic (PLEG): container finished" podID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerID="4bfb25a89190b226ee5311aa3d0a434af6f246af316e1287d1178bbb70b71d3b" exitCode=60 Nov 03 07:45:17.080852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.080743 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" event={"ID":"38515a4d-17c9-45dc-995a-dea48bd883dc","Type":"ContainerDied","Data":"4bfb25a89190b226ee5311aa3d0a434af6f246af316e1287d1178bbb70b71d3b"} Nov 03 07:45:17.080852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.080798 4925 scope.go:122] "RemoveContainer" containerID="eed68d57581ef4871a75832f0902cb16c79947779a88497bcd32c26e1c5f3cbc" Nov 03 07:45:17.081226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.081208 4925 scope.go:122] "RemoveContainer" containerID="4bfb25a89190b226ee5311aa3d0a434af6f246af316e1287d1178bbb70b71d3b" Nov 03 07:45:17.081510 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:17.081482 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7_odh-ai-gateway-infra(38515a4d-17c9-45dc-995a-dea48bd883dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" Nov 03 07:45:17.184091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.184063 4925 scope.go:122] "RemoveContainer" containerID="75660a95c727a303dea5580a9da815dbf0fd885c1fd3bb848f105b3f83feb990" Nov 03 07:45:17.211338 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.211310 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w"] Nov 03 07:45:17.382603 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.382578 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:17.500611 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.500582 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7756b04-4fca-44d7-af02-0bef47d8ae3b-kube-api-access-tftxs\" (UniqueName: \"kubernetes.io/projected/e7756b04-4fca-44d7-af02-0bef47d8ae3b-kube-api-access-tftxs\") pod \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " Nov 03 07:45:17.500799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.500627 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7756b04-4fca-44d7-af02-0bef47d8ae3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7756b04-4fca-44d7-af02-0bef47d8ae3b-maas-api-service-ca\") pod \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\" (UID: \"e7756b04-4fca-44d7-af02-0bef47d8ae3b\") " Nov 03 07:45:17.501845 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.501819 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7756b04-4fca-44d7-af02-0bef47d8ae3b-maas-api-service-ca" pod "e7756b04-4fca-44d7-af02-0bef47d8ae3b" (UID: "e7756b04-4fca-44d7-af02-0bef47d8ae3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:45:17.503338 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.503306 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7756b04-4fca-44d7-af02-0bef47d8ae3b-kube-api-access-tftxs" pod "e7756b04-4fca-44d7-af02-0bef47d8ae3b" (UID: "e7756b04-4fca-44d7-af02-0bef47d8ae3b"). InnerVolumeSpecName "kube-api-access-tftxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:17.560103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.560052 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:17.601942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.601855 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftxs\" (UniqueName: \"kubernetes.io/projected/e7756b04-4fca-44d7-af02-0bef47d8ae3b-kube-api-access-tftxs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:17.601942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:17.601882 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7756b04-4fca-44d7-af02-0bef47d8ae3b-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:18.089497 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.089466 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" Nov 03 07:45:18.089667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.089497 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w" event={"ID":"e7756b04-4fca-44d7-af02-0bef47d8ae3b","Type":"ContainerDied","Data":"4f33eabea42007b68861a3ead2541deca8cd85d54e7078c6527ee7635b852c0e"} Nov 03 07:45:18.089667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.089591 4925 scope.go:122] "RemoveContainer" containerID="bfb8f89c811da0989996d68a7fc63b8385b62333fc06a669452915073ad968ba" Nov 03 07:45:18.091809 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.091774 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerStarted","Data":"27b820fad7f1326599675a54e0bd8cb672bdc6e4c9907570243d8c30526a263e"} Nov 03 07:45:18.183678 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.183653 4925 scope.go:122] "RemoveContainer" containerID="1f726af94d53fc15f246da02a919594a0299d056f0bbebe7b2a9fa636a25d7c3" Nov 03 07:45:18.212889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.212864 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7"] Nov 03 07:45:18.263124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.263090 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w"] Nov 03 07:45:18.265742 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.265715 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-18b89b-29894865-vh56w"] Nov 03 07:45:18.385929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.385904 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:18.511904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.511876 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38515a4d-17c9-45dc-995a-dea48bd883dc-kube-api-access-srnsg\" (UniqueName: \"kubernetes.io/projected/38515a4d-17c9-45dc-995a-dea48bd883dc-kube-api-access-srnsg\") pod \"38515a4d-17c9-45dc-995a-dea48bd883dc\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " Nov 03 07:45:18.512087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.511913 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38515a4d-17c9-45dc-995a-dea48bd883dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38515a4d-17c9-45dc-995a-dea48bd883dc-maas-api-service-ca\") pod \"38515a4d-17c9-45dc-995a-dea48bd883dc\" (UID: \"38515a4d-17c9-45dc-995a-dea48bd883dc\") " Nov 03 07:45:18.512315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.512295 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38515a4d-17c9-45dc-995a-dea48bd883dc-maas-api-service-ca" pod "38515a4d-17c9-45dc-995a-dea48bd883dc" (UID: "38515a4d-17c9-45dc-995a-dea48bd883dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:45:18.512426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.512414 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38515a4d-17c9-45dc-995a-dea48bd883dc-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:18.514234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.514209 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38515a4d-17c9-45dc-995a-dea48bd883dc-kube-api-access-srnsg" pod "38515a4d-17c9-45dc-995a-dea48bd883dc" (UID: "38515a4d-17c9-45dc-995a-dea48bd883dc"). InnerVolumeSpecName "kube-api-access-srnsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:18.613142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.613066 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srnsg\" (UniqueName: \"kubernetes.io/projected/38515a4d-17c9-45dc-995a-dea48bd883dc-kube-api-access-srnsg\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:18.947005 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:18.946921 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:19.102462 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.102428 4925 generic.go:356] "Generic (PLEG): container finished" podID="39e52396-ef05-4008-919f-1610ce4a4aab" containerID="1e4a52efb6c1fcfe3c35ef4d947d08589fce395957daf377a87d766b59df2e9f" exitCode=60 Nov 03 07:45:19.102637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.102504 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" event={"ID":"39e52396-ef05-4008-919f-1610ce4a4aab","Type":"ContainerDied","Data":"1e4a52efb6c1fcfe3c35ef4d947d08589fce395957daf377a87d766b59df2e9f"} Nov 03 07:45:19.102637 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.102575 4925 scope.go:122] "RemoveContainer" containerID="1f726af94d53fc15f246da02a919594a0299d056f0bbebe7b2a9fa636a25d7c3" Nov 03 07:45:19.102969 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.102949 4925 scope.go:122] "RemoveContainer" containerID="1e4a52efb6c1fcfe3c35ef4d947d08589fce395957daf377a87d766b59df2e9f" Nov 03 07:45:19.103317 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:19.103289 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj_odh-ai-gateway-infra(39e52396-ef05-4008-919f-1610ce4a4aab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" Nov 03 07:45:19.105319 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.105299 4925 generic.go:356] "Generic (PLEG): container finished" podID="6687c623-2897-4891-aefe-b9d285910a84" containerID="27b820fad7f1326599675a54e0bd8cb672bdc6e4c9907570243d8c30526a263e" exitCode=60 Nov 03 07:45:19.105433 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.105358 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerDied","Data":"27b820fad7f1326599675a54e0bd8cb672bdc6e4c9907570243d8c30526a263e"} Nov 03 07:45:19.105741 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.105726 4925 scope.go:122] "RemoveContainer" containerID="27b820fad7f1326599675a54e0bd8cb672bdc6e4c9907570243d8c30526a263e" Nov 03 07:45:19.106000 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:19.105977 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb_odh-ai-gateway-infra(6687c623-2897-4891-aefe-b9d285910a84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" podUID="6687c623-2897-4891-aefe-b9d285910a84" Nov 03 07:45:19.106770 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.106750 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" Nov 03 07:45:19.106889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.106775 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7" event={"ID":"38515a4d-17c9-45dc-995a-dea48bd883dc","Type":"ContainerDied","Data":"3d3c6b50e6eb81505f154af10b2017ebe8fde38fb4a87e1d68b4adb32007177d"} Nov 03 07:45:19.119826 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.119805 4925 scope.go:122] "RemoveContainer" containerID="75660a95c727a303dea5580a9da815dbf0fd885c1fd3bb848f105b3f83feb990" Nov 03 07:45:19.133576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.133555 4925 scope.go:122] "RemoveContainer" containerID="4bfb25a89190b226ee5311aa3d0a434af6f246af316e1287d1178bbb70b71d3b" Nov 03 07:45:19.188630 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.188595 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:19.393395 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.393360 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb"] Nov 03 07:45:19.464050 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.463999 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7"] Nov 03 07:45:19.474360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:19.474331 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-44c08f-29894865-98lh7"] Nov 03 07:45:20.193660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.193626 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" path="/var/lib/kubelet/pods/38515a4d-17c9-45dc-995a-dea48bd883dc/volumes" Nov 03 07:45:20.194295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.194272 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" path="/var/lib/kubelet/pods/e7756b04-4fca-44d7-af02-0bef47d8ae3b/volumes" Nov 03 07:45:20.239095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.239059 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj"] Nov 03 07:45:20.296292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.296264 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:20.417764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.417737 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:20.433147 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.433119 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6687c623-2897-4891-aefe-b9d285910a84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6687c623-2897-4891-aefe-b9d285910a84-maas-api-service-ca\") pod \"6687c623-2897-4891-aefe-b9d285910a84\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " Nov 03 07:45:20.433297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.433246 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6687c623-2897-4891-aefe-b9d285910a84-kube-api-access-788q9\" (UniqueName: \"kubernetes.io/projected/6687c623-2897-4891-aefe-b9d285910a84-kube-api-access-788q9\") pod \"6687c623-2897-4891-aefe-b9d285910a84\" (UID: \"6687c623-2897-4891-aefe-b9d285910a84\") " Nov 03 07:45:20.433495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.433465 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6687c623-2897-4891-aefe-b9d285910a84-maas-api-service-ca" pod "6687c623-2897-4891-aefe-b9d285910a84" (UID: "6687c623-2897-4891-aefe-b9d285910a84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:45:20.435405 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.435385 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6687c623-2897-4891-aefe-b9d285910a84-kube-api-access-788q9" pod "6687c623-2897-4891-aefe-b9d285910a84" (UID: "6687c623-2897-4891-aefe-b9d285910a84"). InnerVolumeSpecName "kube-api-access-788q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:20.534944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.534910 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e52396-ef05-4008-919f-1610ce4a4aab-kube-api-access-7wxjb\" (UniqueName: \"kubernetes.io/projected/39e52396-ef05-4008-919f-1610ce4a4aab-kube-api-access-7wxjb\") pod \"39e52396-ef05-4008-919f-1610ce4a4aab\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " Nov 03 07:45:20.535160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.535043 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e52396-ef05-4008-919f-1610ce4a4aab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e52396-ef05-4008-919f-1610ce4a4aab-maas-api-service-ca\") pod \"39e52396-ef05-4008-919f-1610ce4a4aab\" (UID: \"39e52396-ef05-4008-919f-1610ce4a4aab\") " Nov 03 07:45:20.535478 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.535456 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788q9\" (UniqueName: \"kubernetes.io/projected/6687c623-2897-4891-aefe-b9d285910a84-kube-api-access-788q9\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:20.535619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.535480 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6687c623-2897-4891-aefe-b9d285910a84-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:20.535830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.535799 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e52396-ef05-4008-919f-1610ce4a4aab-maas-api-service-ca" pod "39e52396-ef05-4008-919f-1610ce4a4aab" (UID: "39e52396-ef05-4008-919f-1610ce4a4aab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:45:20.538431 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.538404 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e52396-ef05-4008-919f-1610ce4a4aab-kube-api-access-7wxjb" pod "39e52396-ef05-4008-919f-1610ce4a4aab" (UID: "39e52396-ef05-4008-919f-1610ce4a4aab"). InnerVolumeSpecName "kube-api-access-7wxjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:20.637065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.637026 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e52396-ef05-4008-919f-1610ce4a4aab-maas-api-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:20.637065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:20.637068 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wxjb\" (UniqueName: \"kubernetes.io/projected/39e52396-ef05-4008-919f-1610ce4a4aab-kube-api-access-7wxjb\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:21.125661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.125624 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" Nov 03 07:45:21.125661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.125650 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj" event={"ID":"39e52396-ef05-4008-919f-1610ce4a4aab","Type":"ContainerDied","Data":"0caa7d89713f3f036b7b1f4a31242e47b5ac6dc10683283f50a7d1234efea766"} Nov 03 07:45:21.125905 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.125710 4925 scope.go:122] "RemoveContainer" containerID="1e4a52efb6c1fcfe3c35ef4d947d08589fce395957daf377a87d766b59df2e9f" Nov 03 07:45:21.127371 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.127349 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" event={"ID":"6687c623-2897-4891-aefe-b9d285910a84","Type":"ContainerDied","Data":"601b2fedd2fe520841b0a6c089afd3a9561bd15f69930ff3af85e7821bdd8208"} Nov 03 07:45:21.127511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.127375 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb" Nov 03 07:45:21.136226 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.136203 4925 scope.go:122] "RemoveContainer" containerID="27b820fad7f1326599675a54e0bd8cb672bdc6e4c9907570243d8c30526a263e" Nov 03 07:45:21.301562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.301530 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb"] Nov 03 07:45:21.307710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.307686 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e8bf4-29894865-dq7gb"] Nov 03 07:45:21.422270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.422189 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj"] Nov 03 07:45:21.429342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:21.429303 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dcdb23-29894865-cs8rj"] Nov 03 07:45:22.191499 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:22.191465 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" path="/var/lib/kubelet/pods/39e52396-ef05-4008-919f-1610ce4a4aab/volumes" Nov 03 07:45:22.191888 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:22.191873 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6687c623-2897-4891-aefe-b9d285910a84" path="/var/lib/kubelet/pods/6687c623-2897-4891-aefe-b9d285910a84/volumes" Nov 03 07:45:22.568070 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:22.568040 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:26.931170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:26.931132 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:28.493406 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:28.493364 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:28.732833 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:28.732796 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:28.926403 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:28.926364 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:29.627778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:29.627739 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:33.442672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:33.442628 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:34.999606 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:34.999564 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:35.241289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:35.241237 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:35.431298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:35.431259 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:35.980344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:35.980306 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:37.303410 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:37.303377 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:37.526638 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:37.526605 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k"] Nov 03 07:45:37.550824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:37.550781 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:37.761594 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:37.761566 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:38.235981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:38.235922 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:39.138214 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:39.138178 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:45:40.820982 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:40.820932 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:41.462843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:41.462803 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:43.173533 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:43.173493 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7"] Nov 03 07:45:43.308139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:43.308105 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:43.649996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:43.649949 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:43.837661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:43.837616 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:44.073731 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:44.073694 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:44.491123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:44.490999 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:46.181651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:46.181613 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:45:46.352337 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:46.352298 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:49.428548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:49.428512 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w"] Nov 03 07:45:49.616565 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:49.616532 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:50.659633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:50.659599 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:50.856453 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:50.856419 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:51.536157 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:51.536100 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:52.551238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.551199 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" podUID="4506a9e5-97e5-48d6-b3bb-20ca29482c37" containerName="main" containerID="cri-o://44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b" gracePeriod=45 Nov 03 07:45:52.824158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.824133 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:45:52.903489 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903457 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-model-cache\") pod \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " Nov 03 07:45:52.903633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903537 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tmp-dir\") pod \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " Nov 03 07:45:52.903633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903576 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-home\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-home\") pod \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " Nov 03 07:45:52.903633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903609 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tls-certs\" (UniqueName: \"kubernetes.io/secret/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tls-certs\") pod \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " Nov 03 07:45:52.903818 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903649 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4506a9e5-97e5-48d6-b3bb-20ca29482c37-kube-api-access-5w7g6\" (UniqueName: \"kubernetes.io/projected/4506a9e5-97e5-48d6-b3bb-20ca29482c37-kube-api-access-5w7g6\") pod \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " Nov 03 07:45:52.903818 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903728 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-dshm\") pod \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\" (UID: \"4506a9e5-97e5-48d6-b3bb-20ca29482c37\") " Nov 03 07:45:52.903818 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903754 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-model-cache" pod "4506a9e5-97e5-48d6-b3bb-20ca29482c37" (UID: "4506a9e5-97e5-48d6-b3bb-20ca29482c37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:52.903977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903822 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-home" pod "4506a9e5-97e5-48d6-b3bb-20ca29482c37" (UID: "4506a9e5-97e5-48d6-b3bb-20ca29482c37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:52.903977 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.903840 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tmp-dir" pod "4506a9e5-97e5-48d6-b3bb-20ca29482c37" (UID: "4506a9e5-97e5-48d6-b3bb-20ca29482c37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:52.904168 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.904146 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:52.904299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.904178 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:52.904299 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.904192 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:52.905999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.905975 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-dshm" pod "4506a9e5-97e5-48d6-b3bb-20ca29482c37" (UID: "4506a9e5-97e5-48d6-b3bb-20ca29482c37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:52.905999 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.905987 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4506a9e5-97e5-48d6-b3bb-20ca29482c37-kube-api-access-5w7g6" pod "4506a9e5-97e5-48d6-b3bb-20ca29482c37" (UID: "4506a9e5-97e5-48d6-b3bb-20ca29482c37"). InnerVolumeSpecName "kube-api-access-5w7g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:52.906151 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:52.905994 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tls-certs" pod "4506a9e5-97e5-48d6-b3bb-20ca29482c37" (UID: "4506a9e5-97e5-48d6-b3bb-20ca29482c37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:53.004801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.004765 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4506a9e5-97e5-48d6-b3bb-20ca29482c37-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:53.004801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.004792 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7g6\" (UniqueName: \"kubernetes.io/projected/4506a9e5-97e5-48d6-b3bb-20ca29482c37-kube-api-access-5w7g6\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:53.004801 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.004801 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4506a9e5-97e5-48d6-b3bb-20ca29482c37-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:53.357174 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.357134 4925 generic.go:356] "Generic (PLEG): container finished" podID="4506a9e5-97e5-48d6-b3bb-20ca29482c37" containerID="44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b" exitCode=0 Nov 03 07:45:53.357362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.357192 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" Nov 03 07:45:53.357362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.357188 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" event={"ID":"4506a9e5-97e5-48d6-b3bb-20ca29482c37","Type":"ContainerDied","Data":"44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b"} Nov 03 07:45:53.357362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.357309 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k" event={"ID":"4506a9e5-97e5-48d6-b3bb-20ca29482c37","Type":"ContainerDied","Data":"ee5b1ac4a3f0db24484113d4fc90e857f0b3689b1a1538c2c1f83020a2daa2f3"} Nov 03 07:45:53.357362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.357329 4925 scope.go:122] "RemoveContainer" containerID="44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b" Nov 03 07:45:53.370875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.370849 4925 scope.go:122] "RemoveContainer" containerID="44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b" Nov 03 07:45:53.371475 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:45:53.371448 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b\": container with ID starting with 44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b not found: ID does not exist" containerID="44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b" Nov 03 07:45:53.371579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.371490 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b"} err="failed to get container status \"44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b\": rpc error: code = NotFound desc = could not find container \"44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b\": container with ID starting with 44002af2592e179fb1a8add7399e53d9fd453fe5f40eb2563a68606be504be0b not found: ID does not exist" Nov 03 07:45:53.550992 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.550957 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k"] Nov 03 07:45:53.558454 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:53.558424 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0e8bf4/test-model-0e8bf4-kserve-859fd7bbb5-vsg7k"] Nov 03 07:45:54.193719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:54.193686 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4506a9e5-97e5-48d6-b3bb-20ca29482c37" path="/var/lib/kubelet/pods/4506a9e5-97e5-48d6-b3bb-20ca29482c37/volumes" Nov 03 07:45:56.842624 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:56.842588 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:57.079360 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:57.079312 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:45:58.197301 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.197256 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" containerID="cri-o://566230d1c5f8de8e78727b69cd44df0b852a158610fa7a901e38abdd3ec33e30" gracePeriod=45 Nov 03 07:45:58.395155 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.395121 4925 generic.go:356] "Generic (PLEG): container finished" podID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerID="566230d1c5f8de8e78727b69cd44df0b852a158610fa7a901e38abdd3ec33e30" exitCode=0 Nov 03 07:45:58.395328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.395206 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" event={"ID":"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e","Type":"ContainerDied","Data":"566230d1c5f8de8e78727b69cd44df0b852a158610fa7a901e38abdd3ec33e30"} Nov 03 07:45:58.467392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.467321 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:45:58.477341 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477320 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tmp-dir\") pod \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " Nov 03 07:45:58.477498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477364 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tls-certs\") pod \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " Nov 03 07:45:58.477498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477410 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-dshm\") pod \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " Nov 03 07:45:58.477498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477438 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-kube-api-access-68wz9\" (UniqueName: \"kubernetes.io/projected/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-kube-api-access-68wz9\") pod \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " Nov 03 07:45:58.477682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477499 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-model-cache\") pod \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " Nov 03 07:45:58.477682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477607 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tmp-dir" pod "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" (UID: "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.477682 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477636 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-home\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-home\") pod \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\" (UID: \"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e\") " Nov 03 07:45:58.477853 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477829 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-model-cache" pod "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" (UID: "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.477987 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.477958 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-home" pod "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" (UID: "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.478303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.478112 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.478303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.478130 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.478303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.478144 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.479800 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.479773 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tls-certs" pod "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" (UID: "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:58.479907 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.479846 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-kube-api-access-68wz9" pod "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" (UID: "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e"). InnerVolumeSpecName "kube-api-access-68wz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:58.480183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.480154 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-dshm" pod "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" (UID: "7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.579507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.579463 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.579507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.579497 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.579507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:58.579511 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wz9\" (UniqueName: \"kubernetes.io/projected/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e-kube-api-access-68wz9\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:59.402826 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:59.402789 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" Nov 03 07:45:59.403331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:59.402832 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" event={"ID":"7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e","Type":"ContainerDied","Data":"9691f6ddf90c0af2077b44683a7707a2adf0a4bf1927397580a13c975f37fdd3"} Nov 03 07:45:59.403331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:59.402891 4925 scope.go:122] "RemoveContainer" containerID="566230d1c5f8de8e78727b69cd44df0b852a158610fa7a901e38abdd3ec33e30" Nov 03 07:45:59.440931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:59.440898 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:45:59.575199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:59.575164 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7"] Nov 03 07:45:59.578447 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:45:59.578416 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/simulator-w3-1f2e7e-kserve-6678b5cfd4-s7pb7"] Nov 03 07:46:00.192522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:00.192489 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" path="/var/lib/kubelet/pods/7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e/volumes" Nov 03 07:46:01.832631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:01.832584 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:04.454530 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.454478 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" containerID="cri-o://35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae" gracePeriod=45 Nov 03 07:46:04.720827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.720770 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:46:04.746579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.746555 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28910ce8-0366-4962-94a8-4ed5667820a3-kube-api-access-jc85j\" (UniqueName: \"kubernetes.io/projected/28910ce8-0366-4962-94a8-4ed5667820a3-kube-api-access-jc85j\") pod \"28910ce8-0366-4962-94a8-4ed5667820a3\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " Nov 03 07:46:04.746712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.746670 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28910ce8-0366-4962-94a8-4ed5667820a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/28910ce8-0366-4962-94a8-4ed5667820a3-tls-certs\") pod \"28910ce8-0366-4962-94a8-4ed5667820a3\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " Nov 03 07:46:04.746712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.746707 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-model-cache\") pod \"28910ce8-0366-4962-94a8-4ed5667820a3\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " Nov 03 07:46:04.746836 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.746782 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-home\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-home\") pod \"28910ce8-0366-4962-94a8-4ed5667820a3\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " Nov 03 07:46:04.746836 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.746802 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-tmp-dir\") pod \"28910ce8-0366-4962-94a8-4ed5667820a3\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " Nov 03 07:46:04.746943 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.746836 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-dshm\") pod \"28910ce8-0366-4962-94a8-4ed5667820a3\" (UID: \"28910ce8-0366-4962-94a8-4ed5667820a3\") " Nov 03 07:46:04.747229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.747089 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-home" pod "28910ce8-0366-4962-94a8-4ed5667820a3" (UID: "28910ce8-0366-4962-94a8-4ed5667820a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.747229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.747103 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-tmp-dir" pod "28910ce8-0366-4962-94a8-4ed5667820a3" (UID: "28910ce8-0366-4962-94a8-4ed5667820a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.747229 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.747198 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-model-cache" pod "28910ce8-0366-4962-94a8-4ed5667820a3" (UID: "28910ce8-0366-4962-94a8-4ed5667820a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.747437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.747314 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.747437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.747336 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.747437 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.747348 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.748926 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.748906 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28910ce8-0366-4962-94a8-4ed5667820a3-tls-certs" pod "28910ce8-0366-4962-94a8-4ed5667820a3" (UID: "28910ce8-0366-4962-94a8-4ed5667820a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:46:04.749026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.748918 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28910ce8-0366-4962-94a8-4ed5667820a3-kube-api-access-jc85j" pod "28910ce8-0366-4962-94a8-4ed5667820a3" (UID: "28910ce8-0366-4962-94a8-4ed5667820a3"). InnerVolumeSpecName "kube-api-access-jc85j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:04.749096 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.749048 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-dshm" pod "28910ce8-0366-4962-94a8-4ed5667820a3" (UID: "28910ce8-0366-4962-94a8-4ed5667820a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.848476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.848445 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28910ce8-0366-4962-94a8-4ed5667820a3-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.848476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.848479 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc85j\" (UniqueName: \"kubernetes.io/projected/28910ce8-0366-4962-94a8-4ed5667820a3-kube-api-access-jc85j\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.848691 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:04.848494 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28910ce8-0366-4962-94a8-4ed5667820a3-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:05.441067 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.441028 4925 generic.go:356] "Generic (PLEG): container finished" podID="28910ce8-0366-4962-94a8-4ed5667820a3" containerID="35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae" exitCode=0 Nov 03 07:46:05.441216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.441076 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" Nov 03 07:46:05.441216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.441058 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" event={"ID":"28910ce8-0366-4962-94a8-4ed5667820a3","Type":"ContainerDied","Data":"35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae"} Nov 03 07:46:05.441216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.441191 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w" event={"ID":"28910ce8-0366-4962-94a8-4ed5667820a3","Type":"ContainerDied","Data":"a245b87258a28498fbecfd735df7f460de29c7ebf7702bdec9b91069b45b9a13"} Nov 03 07:46:05.441216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.441213 4925 scope.go:122] "RemoveContainer" containerID="35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae" Nov 03 07:46:05.451752 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.451732 4925 scope.go:122] "RemoveContainer" containerID="35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae" Nov 03 07:46:05.452045 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:46:05.452024 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae\": container with ID starting with 35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae not found: ID does not exist" containerID="35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae" Nov 03 07:46:05.452091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.452054 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae"} err="failed to get container status \"35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae\": rpc error: code = NotFound desc = could not find container \"35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae\": container with ID starting with 35651e924100677a0547ec76da95b780bfb1e91c784a8a7d4cdf4d87c7a428ae not found: ID does not exist" Nov 03 07:46:05.614370 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.614332 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w"] Nov 03 07:46:05.665194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:05.665162 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/simulator-w2-669d6b-kserve-96bf6b7fd-whb5w"] Nov 03 07:46:06.190334 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:06.190297 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" path="/var/lib/kubelet/pods/28910ce8-0366-4962-94a8-4ed5667820a3/volumes" Nov 03 07:46:06.848732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:06.848671 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:07.042103 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:07.042068 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:09.139155 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:46:09.139114 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:46:09.531082 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:09.530977 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:09.716727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:09.716678 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:11.852958 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:11.852866 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:12.896461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:12.896416 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:14.371836 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:14.371785 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:16.388732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:16.388689 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:46:17.122335 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:17.122286 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:21.009466 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:21.009428 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:21.774777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:21.774736 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:21.992362 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:21.992319 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:22.185149 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:22.185112 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:22.377844 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:22.377798 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:23.268078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:23.268043 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:25.303836 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:25.303799 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:25.541319 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:25.541287 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:27.310849 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:27.310809 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:27.554275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:27.554233 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:29.163121 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:29.163075 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:30.496389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:30.496358 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:33.690114 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:33.690071 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:36.399697 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:36.399655 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:36.854266 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:36.854232 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv"] Nov 03 07:46:36.854469 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:36.854447 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" podUID="e2a3483b-f650-46e3-a927-002cd5cf94ca" containerName="maas-api" containerID="cri-o://377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67" gracePeriod=30 Nov 03 07:46:37.010350 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.010313 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:37.158058 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.157063 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:46:37.204329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.204296 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls\") pod \"e2a3483b-f650-46e3-a927-002cd5cf94ca\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " Nov 03 07:46:37.204520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.204343 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a3483b-f650-46e3-a927-002cd5cf94ca-kube-api-access-hcm5b\" (UniqueName: \"kubernetes.io/projected/e2a3483b-f650-46e3-a927-002cd5cf94ca-kube-api-access-hcm5b\") pod \"e2a3483b-f650-46e3-a927-002cd5cf94ca\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " Nov 03 07:46:37.204520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.204404 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-metrics-certs\") pod \"e2a3483b-f650-46e3-a927-002cd5cf94ca\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " Nov 03 07:46:37.204520 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.204424 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a3483b-f650-46e3-a927-002cd5cf94ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a3483b-f650-46e3-a927-002cd5cf94ca-openshift-service-ca\") pod \"e2a3483b-f650-46e3-a927-002cd5cf94ca\" (UID: \"e2a3483b-f650-46e3-a927-002cd5cf94ca\") " Nov 03 07:46:37.204911 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.204868 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a3483b-f650-46e3-a927-002cd5cf94ca-openshift-service-ca" pod "e2a3483b-f650-46e3-a927-002cd5cf94ca" (UID: "e2a3483b-f650-46e3-a927-002cd5cf94ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:46:37.207060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.207028 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-metrics-certs" pod "e2a3483b-f650-46e3-a927-002cd5cf94ca" (UID: "e2a3483b-f650-46e3-a927-002cd5cf94ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:46:37.207240 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.207218 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls" pod "e2a3483b-f650-46e3-a927-002cd5cf94ca" (UID: "e2a3483b-f650-46e3-a927-002cd5cf94ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:46:37.207310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.207267 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a3483b-f650-46e3-a927-002cd5cf94ca-kube-api-access-hcm5b" pod "e2a3483b-f650-46e3-a927-002cd5cf94ca" (UID: "e2a3483b-f650-46e3-a927-002cd5cf94ca"). InnerVolumeSpecName "kube-api-access-hcm5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:37.315573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.306423 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:37.315573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.310186 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a3483b-f650-46e3-a927-002cd5cf94ca-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:37.315573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.310225 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a3483b-f650-46e3-a927-002cd5cf94ca-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:37.315573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.310243 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcm5b\" (UniqueName: \"kubernetes.io/projected/e2a3483b-f650-46e3-a927-002cd5cf94ca-kube-api-access-hcm5b\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:37.666916 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.666876 4925 generic.go:356] "Generic (PLEG): container finished" podID="e2a3483b-f650-46e3-a927-002cd5cf94ca" containerID="377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67" exitCode=0 Nov 03 07:46:37.667393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.666952 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" Nov 03 07:46:37.667393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.666968 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" event={"ID":"e2a3483b-f650-46e3-a927-002cd5cf94ca","Type":"ContainerDied","Data":"377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67"} Nov 03 07:46:37.667393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.667081 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv" event={"ID":"e2a3483b-f650-46e3-a927-002cd5cf94ca","Type":"ContainerDied","Data":"6d83e0d1e648e27fe6b1d66fbae826552b88a44ad71cf15c9d5be98f8d8c3de6"} Nov 03 07:46:37.667393 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.667112 4925 scope.go:122] "RemoveContainer" containerID="377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67" Nov 03 07:46:37.678908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.678882 4925 scope.go:122] "RemoveContainer" containerID="377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67" Nov 03 07:46:37.679235 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:46:37.679207 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67\": container with ID starting with 377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67 not found: ID does not exist" containerID="377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67" Nov 03 07:46:37.679328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.679250 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67"} err="failed to get container status \"377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67\": rpc error: code = NotFound desc = could not find container \"377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67\": container with ID starting with 377aa2f01c4da367dd926635ba637f7756b3597ff4d164ee9e4dcc7c808a0b67 not found: ID does not exist" Nov 03 07:46:37.844211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.844168 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv"] Nov 03 07:46:37.893890 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:37.893856 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1f2e7e-768c658d-2w7dv"] Nov 03 07:46:38.193062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:38.192998 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a3483b-f650-46e3-a927-002cd5cf94ca" path="/var/lib/kubelet/pods/e2a3483b-f650-46e3-a927-002cd5cf94ca/volumes" Nov 03 07:46:38.199451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:38.199421 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:38.572176 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:38.572140 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:38.860919 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:38.860825 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:39.139111 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:46:39.139023 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:46:39.545924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:39.545835 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:39.839461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:39.839425 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:40.195148 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:40.195073 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:40.405212 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:40.405183 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:40.893660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:40.893621 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:45.624551 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:45.624512 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:47.490789 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:47.490739 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:47.681291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:47.681250 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:47.863688 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:47.863652 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:49.116082 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:49.116049 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:49.308774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:49.308736 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:49.500351 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:49.500149 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:50.790204 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:50.790106 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:55.761443 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:55.761407 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:46:56.243852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:56.243814 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:59.278848 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:59.278805 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:46:59.472739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:46:59.472701 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:01.537356 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:01.537315 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:02.310944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:02.310895 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:02.504620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:02.504531 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:03.973095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:03.973058 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:04.161853 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:04.161814 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:05.938655 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:05.938613 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:07.202640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:07.202608 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:07.450961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:07.450920 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:07.654863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:07.654826 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:07.869904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:07.869871 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:08.464262 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:08.464219 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:09.138512 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:09.138480 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:47:13.038709 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:13.038657 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:15.583946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:15.583907 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:15.882505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:15.882420 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:15.886448 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:15.886416 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk"] Nov 03 07:47:15.886751 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:15.886705 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" containerID="cri-o://203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be" gracePeriod=30 Nov 03 07:47:16.073234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:16.073201 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:18.049472 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:18.049427 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:20.414933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:20.414900 4925 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:47:20.415418 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:20.414968 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:47:21.154054 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.153979 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:47:21.194216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194193 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-data\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194248 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-certs\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194315 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-podinfo\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194367 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194350 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-socket\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194388 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f264c5c8-6531-4981-bafe-61609a4a60fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f264c5c8-6531-4981-bafe-61609a4a60fa-istiod-ca-cert\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194419 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-envoy\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194461 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-token\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-token\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194489 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-credential-socket\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194499 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-data" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:21.194548 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194545 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-kube-api-access-b2xcx\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-kube-api-access-b2xcx\") pod \"f264c5c8-6531-4981-bafe-61609a4a60fa\" (UID: \"f264c5c8-6531-4981-bafe-61609a4a60fa\") " Nov 03 07:47:21.194872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194593 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-certs" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:21.194872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194630 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-socket" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:21.194872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194811 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-credential-socket" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:21.194872 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194820 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f264c5c8-6531-4981-bafe-61609a4a60fa-istiod-ca-cert" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:21.195043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194981 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.195043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.194998 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.195043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.195035 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.195145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.195046 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f264c5c8-6531-4981-bafe-61609a4a60fa-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.195145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.195055 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.196610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.196589 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-token" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:21.197087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.197063 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-kube-api-access-b2xcx" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "kube-api-access-b2xcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:21.197191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.197128 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-podinfo" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:47:21.197191 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.197146 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-envoy" pod "f264c5c8-6531-4981-bafe-61609a4a60fa" (UID: "f264c5c8-6531-4981-bafe-61609a4a60fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:21.295765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.295739 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.295765 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.295765 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.295929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.295776 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xcx\" (UniqueName: \"kubernetes.io/projected/f264c5c8-6531-4981-bafe-61609a4a60fa-kube-api-access-b2xcx\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.295929 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.295786 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f264c5c8-6531-4981-bafe-61609a4a60fa-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:21.943805 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.943761 4925 generic.go:356] "Generic (PLEG): container finished" podID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerID="203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be" exitCode=0 Nov 03 07:47:21.944309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.943810 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" event={"ID":"f264c5c8-6531-4981-bafe-61609a4a60fa","Type":"ContainerDied","Data":"203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be"} Nov 03 07:47:21.944309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.943821 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" Nov 03 07:47:21.944309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.943875 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk" event={"ID":"f264c5c8-6531-4981-bafe-61609a4a60fa","Type":"ContainerDied","Data":"01bc86045cf83c92bc13f991a61f33d2701deac48d6a8697a80ce799eabe5d2b"} Nov 03 07:47:21.944309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.943895 4925 scope.go:122] "RemoveContainer" containerID="203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be" Nov 03 07:47:21.954984 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.954965 4925 scope.go:122] "RemoveContainer" containerID="203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be" Nov 03 07:47:21.955276 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:21.955258 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be\": container with ID starting with 203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be not found: ID does not exist" containerID="203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be" Nov 03 07:47:21.955330 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:21.955283 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be"} err="failed to get container status \"203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be\": rpc error: code = NotFound desc = could not find container \"203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be\": container with ID starting with 203e9d9a97cf6c1c3db7e48d4e1dd0ed6fdf009e88c2ccd264c911939c7530be not found: ID does not exist" Nov 03 07:47:22.116666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.116632 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk"] Nov 03 07:47:22.167455 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.167425 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk"] Nov 03 07:47:22.190623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.190593 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" path="/var/lib/kubelet/pods/f264c5c8-6531-4981-bafe-61609a4a60fa/volumes" Nov 03 07:47:22.394060 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.394026 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn"] Nov 03 07:47:22.394282 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.394256 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" podUID="f74065d0-b412-4763-8da5-9ec45c612370" containerName="maas-api" containerID="cri-o://38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18" gracePeriod=30 Nov 03 07:47:22.517257 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.517226 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:22.683449 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.683413 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:47:22.710440 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.710417 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-maas-api-tls\") pod \"f74065d0-b412-4763-8da5-9ec45c612370\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " Nov 03 07:47:22.710593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.710469 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74065d0-b412-4763-8da5-9ec45c612370-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74065d0-b412-4763-8da5-9ec45c612370-openshift-service-ca\") pod \"f74065d0-b412-4763-8da5-9ec45c612370\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " Nov 03 07:47:22.710593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.710529 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74065d0-b412-4763-8da5-9ec45c612370-kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/f74065d0-b412-4763-8da5-9ec45c612370-kube-api-access-x2kkr\") pod \"f74065d0-b412-4763-8da5-9ec45c612370\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " Nov 03 07:47:22.710593 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.710581 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-metrics-certs\") pod \"f74065d0-b412-4763-8da5-9ec45c612370\" (UID: \"f74065d0-b412-4763-8da5-9ec45c612370\") " Nov 03 07:47:22.710886 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.710859 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74065d0-b412-4763-8da5-9ec45c612370-openshift-service-ca" pod "f74065d0-b412-4763-8da5-9ec45c612370" (UID: "f74065d0-b412-4763-8da5-9ec45c612370"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:22.712875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.712849 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-maas-api-tls" pod "f74065d0-b412-4763-8da5-9ec45c612370" (UID: "f74065d0-b412-4763-8da5-9ec45c612370"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:22.712972 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.712870 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-metrics-certs" pod "f74065d0-b412-4763-8da5-9ec45c612370" (UID: "f74065d0-b412-4763-8da5-9ec45c612370"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:22.712972 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.712911 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74065d0-b412-4763-8da5-9ec45c612370-kube-api-access-x2kkr" pod "f74065d0-b412-4763-8da5-9ec45c612370" (UID: "f74065d0-b412-4763-8da5-9ec45c612370"). InnerVolumeSpecName "kube-api-access-x2kkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:22.812561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.812529 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:22.812561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.812558 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f74065d0-b412-4763-8da5-9ec45c612370-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:22.812561 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.812568 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74065d0-b412-4763-8da5-9ec45c612370-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:22.812798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.812578 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/f74065d0-b412-4763-8da5-9ec45c612370-kube-api-access-x2kkr\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:22.954125 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.954038 4925 generic.go:356] "Generic (PLEG): container finished" podID="f74065d0-b412-4763-8da5-9ec45c612370" containerID="38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18" exitCode=0 Nov 03 07:47:22.954566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.954135 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" Nov 03 07:47:22.954566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.954152 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" event={"ID":"f74065d0-b412-4763-8da5-9ec45c612370","Type":"ContainerDied","Data":"38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18"} Nov 03 07:47:22.954566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.954208 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn" event={"ID":"f74065d0-b412-4763-8da5-9ec45c612370","Type":"ContainerDied","Data":"f71b072b9f6e27ff6f9b49769234c102059d80ef90e20a1caa60a05086b06f79"} Nov 03 07:47:22.954566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.954242 4925 scope.go:122] "RemoveContainer" containerID="38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18" Nov 03 07:47:22.964732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.964703 4925 scope.go:122] "RemoveContainer" containerID="38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18" Nov 03 07:47:22.964968 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:22.964951 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18\": container with ID starting with 38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18 not found: ID does not exist" containerID="38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18" Nov 03 07:47:22.965119 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:22.964978 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18"} err="failed to get container status \"38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18\": rpc error: code = NotFound desc = could not find container \"38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18\": container with ID starting with 38ec8f71e748f41ec4ac927082ce843c5f0d9fa8f88a00b0e9c89892fc186b18 not found: ID does not exist" Nov 03 07:47:23.127746 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:23.127708 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn"] Nov 03 07:47:23.132352 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:23.132325 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcdb23-c7d86fcbc-vkgpn"] Nov 03 07:47:24.099815 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:24.099776 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:24.190418 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:24.190381 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74065d0-b412-4763-8da5-9ec45c612370" path="/var/lib/kubelet/pods/f74065d0-b412-4763-8da5-9ec45c612370/volumes" Nov 03 07:47:24.329345 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:24.329306 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:24.539897 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:24.539817 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:25.045562 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:25.045526 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:29.386006 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:29.385968 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb"] Nov 03 07:47:29.469891 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:29.469024 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:29.663799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:29.663705 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:31.072586 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:31.072549 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:47:32.524949 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:32.524911 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:32.718448 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:32.718414 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:32.833986 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:32.833949 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4"] Nov 03 07:47:32.834230 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:32.834204 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" containerID="cri-o://d469f17fc022c50332d68eae35b3ead7322fb164d620f46a9cde7c7219f42185" gracePeriod=30 Nov 03 07:47:32.924450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:32.924413 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:33.424399 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:33.424363 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:34.539568 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:34.539530 4925 patch_prober.go:29] interesting pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:47:34.540047 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:34.539594 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:47:34.918953 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:34.918914 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:38.057061 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.057005 4925 generic.go:356] "Generic (PLEG): container finished" podID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerID="d469f17fc022c50332d68eae35b3ead7322fb164d620f46a9cde7c7219f42185" exitCode=0 Nov 03 07:47:38.058909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.058766 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" event={"ID":"a52ae06b-ed08-47b0-be5f-2e522283b659","Type":"ContainerDied","Data":"d469f17fc022c50332d68eae35b3ead7322fb164d620f46a9cde7c7219f42185"} Nov 03 07:47:38.058909 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.058844 4925 scope.go:122] "RemoveContainer" containerID="0a9cae86b01ff5f8b9c989c760d7a0b9eb486aafe25d6a803538dc26c0f671ab" Nov 03 07:47:38.119795 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.119744 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:47:38.284461 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284425 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-credential-socket\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284481 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-podinfo\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284538 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-certs\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284595 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-kube-api-access-c9tds\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-kube-api-access-c9tds\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284623 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52ae06b-ed08-47b0-be5f-2e522283b659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52ae06b-ed08-47b0-be5f-2e522283b659-istiod-ca-cert\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284672 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284663 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-socket\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284705 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-credential-socket" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:38.284985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284751 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-envoy\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284782 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-data\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284812 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-token\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-token\") pod \"a52ae06b-ed08-47b0-be5f-2e522283b659\" (UID: \"a52ae06b-ed08-47b0-be5f-2e522283b659\") " Nov 03 07:47:38.284985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284913 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-certs" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:38.285271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.284984 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52ae06b-ed08-47b0-be5f-2e522283b659-istiod-ca-cert" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:38.285271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.285128 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-data" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:38.285271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.285263 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52ae06b-ed08-47b0-be5f-2e522283b659-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.285432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.285283 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.285432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.285300 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.285432 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.285314 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.285604 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.285537 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-socket" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:38.287646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.287619 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-podinfo" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:47:38.287646 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.287636 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-token" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:38.290301 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.290275 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-kube-api-access-c9tds" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "kube-api-access-c9tds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:38.290436 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.290420 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-envoy" pod "a52ae06b-ed08-47b0-be5f-2e522283b659" (UID: "a52ae06b-ed08-47b0-be5f-2e522283b659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:38.352934 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.352890 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk"] Nov 03 07:47:38.356809 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.356781 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8"] Nov 03 07:47:38.358998 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.358968 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s"] Nov 03 07:47:38.386260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.386196 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.386260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.386223 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.386260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.386235 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52ae06b-ed08-47b0-be5f-2e522283b659-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.386260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.386244 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tds\" (UniqueName: \"kubernetes.io/projected/a52ae06b-ed08-47b0-be5f-2e522283b659-kube-api-access-c9tds\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.386260 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.386254 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52ae06b-ed08-47b0-be5f-2e522283b659-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:38.481524 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:38.481479 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:39.067303 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:39.067270 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" Nov 03 07:47:39.067734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:39.067316 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4" event={"ID":"a52ae06b-ed08-47b0-be5f-2e522283b659","Type":"ContainerDied","Data":"ccf15ac01cf9b6ae08dacfadadf6c8091f73e6509e98850ef12e777e06e3c5ab"} Nov 03 07:47:39.067734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:39.067380 4925 scope.go:122] "RemoveContainer" containerID="d469f17fc022c50332d68eae35b3ead7322fb164d620f46a9cde7c7219f42185" Nov 03 07:47:39.137779 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:39.137748 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:47:39.242342 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:39.242305 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4"] Nov 03 07:47:39.291571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:39.291532 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4"] Nov 03 07:47:40.191031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:40.190981 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" path="/var/lib/kubelet/pods/a52ae06b-ed08-47b0-be5f-2e522283b659/volumes" Nov 03 07:47:40.228924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:40.228874 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:41.642298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:41.642256 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:43.249327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:43.249292 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57"] Nov 03 07:47:43.249875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:43.249561 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" containerID="cri-o://6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e" gracePeriod=30 Nov 03 07:47:43.299408 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:43.299363 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn"] Nov 03 07:47:43.299658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:43.299629 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" containerID="cri-o://5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4" gracePeriod=30 Nov 03 07:47:43.301843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:43.301814 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h"] Nov 03 07:47:43.302071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:43.302045 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" containerID="cri-o://3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084" gracePeriod=30 Nov 03 07:47:44.264467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.264423 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:47:44.406668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.406630 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" podUID="ac12005a-d91c-404f-ab8c-5c94f723b2fc" containerName="main" containerID="cri-o://263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d" gracePeriod=45 Nov 03 07:47:44.668868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.668833 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:47:44.852726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.852690 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tls-certs\") pod \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " Nov 03 07:47:44.852910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.852767 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tmp-dir\") pod \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " Nov 03 07:47:44.852910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.852821 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-home\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-home\") pod \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " Nov 03 07:47:44.852910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.852842 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-model-cache\") pod \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " Nov 03 07:47:44.852910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.852862 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac12005a-d91c-404f-ab8c-5c94f723b2fc-kube-api-access-7cjnq\" (UniqueName: \"kubernetes.io/projected/ac12005a-d91c-404f-ab8c-5c94f723b2fc-kube-api-access-7cjnq\") pod \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " Nov 03 07:47:44.852910 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.852877 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-dshm\") pod \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\" (UID: \"ac12005a-d91c-404f-ab8c-5c94f723b2fc\") " Nov 03 07:47:44.853233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.853111 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-home" pod "ac12005a-d91c-404f-ab8c-5c94f723b2fc" (UID: "ac12005a-d91c-404f-ab8c-5c94f723b2fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:44.853233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.853127 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tmp-dir" pod "ac12005a-d91c-404f-ab8c-5c94f723b2fc" (UID: "ac12005a-d91c-404f-ab8c-5c94f723b2fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:44.853233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.853136 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-model-cache" pod "ac12005a-d91c-404f-ab8c-5c94f723b2fc" (UID: "ac12005a-d91c-404f-ab8c-5c94f723b2fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:44.853390 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.853255 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:44.853390 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.853270 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:44.853390 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.853298 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:44.855366 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.855338 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tls-certs" pod "ac12005a-d91c-404f-ab8c-5c94f723b2fc" (UID: "ac12005a-d91c-404f-ab8c-5c94f723b2fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:44.855505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.855428 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-dshm" pod "ac12005a-d91c-404f-ab8c-5c94f723b2fc" (UID: "ac12005a-d91c-404f-ab8c-5c94f723b2fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:44.855505 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.855493 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac12005a-d91c-404f-ab8c-5c94f723b2fc-kube-api-access-7cjnq" pod "ac12005a-d91c-404f-ab8c-5c94f723b2fc" (UID: "ac12005a-d91c-404f-ab8c-5c94f723b2fc"). InnerVolumeSpecName "kube-api-access-7cjnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:44.953850 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.953821 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjnq\" (UniqueName: \"kubernetes.io/projected/ac12005a-d91c-404f-ab8c-5c94f723b2fc-kube-api-access-7cjnq\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:44.953850 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.953849 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac12005a-d91c-404f-ab8c-5c94f723b2fc-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:44.954042 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:44.953859 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac12005a-d91c-404f-ab8c-5c94f723b2fc-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:45.106217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.106133 4925 generic.go:356] "Generic (PLEG): container finished" podID="ac12005a-d91c-404f-ab8c-5c94f723b2fc" containerID="263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d" exitCode=0 Nov 03 07:47:45.106217 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.106177 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" Nov 03 07:47:45.106412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.106220 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" event={"ID":"ac12005a-d91c-404f-ab8c-5c94f723b2fc","Type":"ContainerDied","Data":"263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d"} Nov 03 07:47:45.106412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.106264 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb" event={"ID":"ac12005a-d91c-404f-ab8c-5c94f723b2fc","Type":"ContainerDied","Data":"88a86e2a55dce804d21e0816bb32f88b09e300a94962da335669842e1ae24671"} Nov 03 07:47:45.106412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.106283 4925 scope.go:122] "RemoveContainer" containerID="263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d" Nov 03 07:47:45.116289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.116271 4925 scope.go:122] "RemoveContainer" containerID="263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d" Nov 03 07:47:45.116556 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:45.116537 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d\": container with ID starting with 263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d not found: ID does not exist" containerID="263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d" Nov 03 07:47:45.116615 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.116566 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d"} err="failed to get container status \"263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d\": rpc error: code = NotFound desc = could not find container \"263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d\": container with ID starting with 263da6387fda6b4f1c733288cb92830ff85cecdc00d7726d560def8135020e5d not found: ID does not exist" Nov 03 07:47:45.256833 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.256793 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf"] Nov 03 07:47:45.279160 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.279133 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb"] Nov 03 07:47:45.288380 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.288348 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-50ae96/test-model-50ae96-kserve-5968d5fcdc-rvlrb"] Nov 03 07:47:45.321564 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.321530 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:45.579821 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:45.579789 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:46.190778 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:46.190740 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac12005a-d91c-404f-ab8c-5c94f723b2fc" path="/var/lib/kubelet/pods/ac12005a-d91c-404f-ab8c-5c94f723b2fc/volumes" Nov 03 07:47:48.584323 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.584291 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:47:48.593279 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593254 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-kube-api-access-7h2xk\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-kube-api-access-7h2xk\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593349 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-token\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-token\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593384 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istiod-ca-cert\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593418 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-envoy\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593588 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593467 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-data\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593588 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593532 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-socket\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593590 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-podinfo\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593635 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-credential-socket\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593679 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-certs\") pod \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\" (UID: \"7f65cfe2-7f4e-40d9-9fec-96b82223194c\") " Nov 03 07:47:48.593902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.593873 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-data" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.594227 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.594170 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.594773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.594582 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-certs" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.594773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.594671 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-socket" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.594773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.594726 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-credential-socket" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.595170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.595147 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istiod-ca-cert" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:48.598824 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.598797 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-podinfo" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:47:48.599245 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.599221 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-kube-api-access-7h2xk" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "kube-api-access-7h2xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.600767 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.600735 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-envoy" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.604224 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.604205 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-token" pod "7f65cfe2-7f4e-40d9-9fec-96b82223194c" (UID: "7f65cfe2-7f4e-40d9-9fec-96b82223194c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.627721 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.627702 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:47:48.629886 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.629869 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:47:48.694668 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.694628 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-certs\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.694863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.694696 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-data\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.694863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.694732 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-istio-token\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-istio-token\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.694863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.694765 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6802fb28-f1af-4a5a-b353-eb333060b469-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6802fb28-f1af-4a5a-b353-eb333060b469-istiod-ca-cert\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.694863 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.694803 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-envoy\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.695066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.694952 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-credential-socket\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.695066 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695004 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-data\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695063 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-kube-api-access-b854j\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-kube-api-access-b854j\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695137 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695112 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-kube-api-access-pp5kv\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-kube-api-access-pp5kv\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.695218 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695148 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-token\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-token\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695267 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695226 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-socket\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.695324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695267 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6802fb28-f1af-4a5a-b353-eb333060b469-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6802fb28-f1af-4a5a-b353-eb333060b469-istio-podinfo\") pod \"6802fb28-f1af-4a5a-b353-eb333060b469\" (UID: \"6802fb28-f1af-4a5a-b353-eb333060b469\") " Nov 03 07:47:48.695324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.694980 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-certs" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.695324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695071 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-data" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.695324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695265 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-data" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.695324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695318 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istiod-ca-cert\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695391 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-certs\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695437 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-credential-socket\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695467 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-podinfo\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695500 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-socket\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695528 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-envoy\") pod \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\" (UID: \"a571841e-f2f4-414a-a6b4-cb3d40f0ad29\") " Nov 03 07:47:48.695961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695905 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.695961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695932 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.695961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695947 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.695961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695962 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.695961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695976 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f65cfe2-7f4e-40d9-9fec-96b82223194c-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.695961 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.695989 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.696404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.696024 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.696404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.696040 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f65cfe2-7f4e-40d9-9fec-96b82223194c-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.696404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.696055 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.696404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.696069 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2xk\" (UniqueName: \"kubernetes.io/projected/7f65cfe2-7f4e-40d9-9fec-96b82223194c-kube-api-access-7h2xk\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.696404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.696084 4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-data\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.696404 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.696372 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-certs" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.696729 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.696564 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-credential-socket" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.697571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.697102 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6802fb28-f1af-4a5a-b353-eb333060b469-istiod-ca-cert" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:48.697571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.697156 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istiod-ca-cert" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:48.697571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.697160 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-socket" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.697571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.697445 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-credential-socket" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.697852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.697640 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-socket" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.698152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.698126 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-envoy" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.698699 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.698667 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-envoy" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.699501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.699374 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-kube-api-access-b854j" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "kube-api-access-b854j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.699501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.699447 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-token" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.699732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.699677 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6802fb28-f1af-4a5a-b353-eb333060b469-istio-podinfo" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:47:48.699965 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.699939 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-kube-api-access-pp5kv" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "kube-api-access-pp5kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.700773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.700752 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-istio-token" pod "6802fb28-f1af-4a5a-b353-eb333060b469" (UID: "6802fb28-f1af-4a5a-b353-eb333060b469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.701281 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.701265 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-podinfo" pod "a571841e-f2f4-414a-a6b4-cb3d40f0ad29" (UID: "a571841e-f2f4-414a-a6b4-cb3d40f0ad29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:47:48.797525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797489 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797518 4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797528 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797537 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797546 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797554 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797563 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797570 4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6802fb28-f1af-4a5a-b353-eb333060b469-istiod-ca-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797578 4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-istio-envoy\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797586 4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-credential-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797594 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b854j\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-kube-api-access-b854j\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797603 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5kv\" (UniqueName: \"kubernetes.io/projected/6802fb28-f1af-4a5a-b353-eb333060b469-kube-api-access-pp5kv\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797612 4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571841e-f2f4-414a-a6b4-cb3d40f0ad29-istio-token\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797621 4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6802fb28-f1af-4a5a-b353-eb333060b469-workload-socket\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.797739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:48.797630 4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6802fb28-f1af-4a5a-b353-eb333060b469-istio-podinfo\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:49.131507 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.131470 4925 generic.go:356] "Generic (PLEG): container finished" podID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerID="6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e" exitCode=0 Nov 03 07:47:49.131717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.131531 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" Nov 03 07:47:49.131717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.131561 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" event={"ID":"7f65cfe2-7f4e-40d9-9fec-96b82223194c","Type":"ContainerDied","Data":"6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e"} Nov 03 07:47:49.131717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.131633 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57" event={"ID":"7f65cfe2-7f4e-40d9-9fec-96b82223194c","Type":"ContainerDied","Data":"7f5424d466a50d7ac6a2a9ec22b0ed40e62228f5cc185ec7a2c485c1c7579ad8"} Nov 03 07:47:49.131717 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.131665 4925 scope.go:122] "RemoveContainer" containerID="6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e" Nov 03 07:47:49.140707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.140678 4925 generic.go:356] "Generic (PLEG): container finished" podID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerID="5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4" exitCode=0 Nov 03 07:47:49.140933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.140892 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" Nov 03 07:47:49.141525 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.141480 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" event={"ID":"a571841e-f2f4-414a-a6b4-cb3d40f0ad29","Type":"ContainerDied","Data":"5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4"} Nov 03 07:47:49.141614 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.141528 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn" event={"ID":"a571841e-f2f4-414a-a6b4-cb3d40f0ad29","Type":"ContainerDied","Data":"3a144e90301492f6f892f9bd7f06cad51d4fff3969a31998d99c4486ba78750e"} Nov 03 07:47:49.143088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.143067 4925 generic.go:356] "Generic (PLEG): container finished" podID="6802fb28-f1af-4a5a-b353-eb333060b469" containerID="3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084" exitCode=0 Nov 03 07:47:49.143195 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.143105 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" Nov 03 07:47:49.143195 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.143122 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" event={"ID":"6802fb28-f1af-4a5a-b353-eb333060b469","Type":"ContainerDied","Data":"3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084"} Nov 03 07:47:49.143195 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.143163 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h" event={"ID":"6802fb28-f1af-4a5a-b353-eb333060b469","Type":"ContainerDied","Data":"4e6023db45cfb9e5ab9683998570dc81b14b9c7dc075023bee7161525dee0796"} Nov 03 07:47:49.151933 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.151919 4925 scope.go:122] "RemoveContainer" containerID="6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e" Nov 03 07:47:49.152393 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:49.152366 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e\": container with ID starting with 6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e not found: ID does not exist" containerID="6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e" Nov 03 07:47:49.152477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.152401 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e"} err="failed to get container status \"6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e\": rpc error: code = NotFound desc = could not find container \"6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e\": container with ID starting with 6c7eae1ee61753005dd75331230e94f8bf265b54eee7db174e2285cb3d238e7e not found: ID does not exist" Nov 03 07:47:49.152477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.152425 4925 scope.go:122] "RemoveContainer" containerID="5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4" Nov 03 07:47:49.161592 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.161572 4925 scope.go:122] "RemoveContainer" containerID="5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4" Nov 03 07:47:49.161864 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:49.161846 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4\": container with ID starting with 5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4 not found: ID does not exist" containerID="5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4" Nov 03 07:47:49.161908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.161872 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4"} err="failed to get container status \"5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4\": rpc error: code = NotFound desc = could not find container \"5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4\": container with ID starting with 5d1e8e395e46e54583f88813e2234e1af790cf6bc6a3e863b288075b1e65dda4 not found: ID does not exist" Nov 03 07:47:49.161908 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.161893 4925 scope.go:122] "RemoveContainer" containerID="3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084" Nov 03 07:47:49.170843 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.170814 4925 scope.go:122] "RemoveContainer" containerID="3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084" Nov 03 07:47:49.171138 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:49.171117 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084\": container with ID starting with 3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084 not found: ID does not exist" containerID="3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084" Nov 03 07:47:49.171212 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.171145 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084"} err="failed to get container status \"3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084\": rpc error: code = NotFound desc = could not find container \"3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084\": container with ID starting with 3b47f0ac7888faf3ae184e84c7018b05efcac67307444f97ed1ba5580b4b7084 not found: ID does not exist" Nov 03 07:47:49.303153 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.303119 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn"] Nov 03 07:47:49.305695 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.305669 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn"] Nov 03 07:47:49.424641 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.424558 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h"] Nov 03 07:47:49.434653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.434620 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h"] Nov 03 07:47:49.535074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.534560 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:49.549803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.549769 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57"] Nov 03 07:47:49.558511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.558484 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57"] Nov 03 07:47:49.729915 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:49.729811 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:50.190198 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:50.190161 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" path="/var/lib/kubelet/pods/6802fb28-f1af-4a5a-b353-eb333060b469/volumes" Nov 03 07:47:50.190608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:50.190594 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" path="/var/lib/kubelet/pods/7f65cfe2-7f4e-40d9-9fec-96b82223194c/volumes" Nov 03 07:47:50.190981 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:50.190969 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" path="/var/lib/kubelet/pods/a571841e-f2f4-414a-a6b4-cb3d40f0ad29/volumes" Nov 03 07:47:50.927671 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:50.927633 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:53.376528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.376489 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" containerID="cri-o://0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349" gracePeriod=45 Nov 03 07:47:53.378255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.378223 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" containerID="cri-o://0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda" gracePeriod=45 Nov 03 07:47:53.381452 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.381177 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" containerID="cri-o://d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0" gracePeriod=45 Nov 03 07:47:53.428015 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.427979 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused" Nov 03 07:47:53.449133 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.449104 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused" Nov 03 07:47:53.480782 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.480755 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": dial tcp 10.132.0.67:8000: connect: connection refused" Nov 03 07:47:53.657100 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.657075 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:47:53.694090 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.694070 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:47:53.700622 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.700602 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:47:53.746450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746424 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-home\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-home\") pod \"a925815c-373f-4571-ba1a-0a0fbc93674a\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " Nov 03 07:47:53.746599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746466 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-model-cache\") pod \"28861b2d-0779-4b84-bfff-76111f6068cd\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " Nov 03 07:47:53.746599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746496 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a925815c-373f-4571-ba1a-0a0fbc93674a-kube-api-access-5qpz9\" (UniqueName: \"kubernetes.io/projected/a925815c-373f-4571-ba1a-0a0fbc93674a-kube-api-access-5qpz9\") pod \"a925815c-373f-4571-ba1a-0a0fbc93674a\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " Nov 03 07:47:53.746599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746525 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-dshm\") pod \"a925815c-373f-4571-ba1a-0a0fbc93674a\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " Nov 03 07:47:53.746599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746554 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a925815c-373f-4571-ba1a-0a0fbc93674a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a925815c-373f-4571-ba1a-0a0fbc93674a-tls-certs\") pod \"a925815c-373f-4571-ba1a-0a0fbc93674a\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " Nov 03 07:47:53.746599 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746578 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-tmp-dir\") pod \"a28a39a6-785f-4b41-91ea-af2ac05b443f\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " Nov 03 07:47:53.746882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746611 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-tmp-dir\") pod \"28861b2d-0779-4b84-bfff-76111f6068cd\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " Nov 03 07:47:53.746882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746668 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a28a39a6-785f-4b41-91ea-af2ac05b443f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a28a39a6-785f-4b41-91ea-af2ac05b443f-tls-certs\") pod \"a28a39a6-785f-4b41-91ea-af2ac05b443f\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " Nov 03 07:47:53.746882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746702 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28861b2d-0779-4b84-bfff-76111f6068cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/28861b2d-0779-4b84-bfff-76111f6068cd-tls-certs\") pod \"28861b2d-0779-4b84-bfff-76111f6068cd\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " Nov 03 07:47:53.746882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746724 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28861b2d-0779-4b84-bfff-76111f6068cd-kube-api-access-6v2lk\" (UniqueName: \"kubernetes.io/projected/28861b2d-0779-4b84-bfff-76111f6068cd-kube-api-access-6v2lk\") pod \"28861b2d-0779-4b84-bfff-76111f6068cd\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " Nov 03 07:47:53.746882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746722 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-home" pod "a925815c-373f-4571-ba1a-0a0fbc93674a" (UID: "a925815c-373f-4571-ba1a-0a0fbc93674a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.746882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746740 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-model-cache\") pod \"a925815c-373f-4571-ba1a-0a0fbc93674a\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " Nov 03 07:47:53.746882 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746831 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-dshm\") pod \"a28a39a6-785f-4b41-91ea-af2ac05b443f\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " Nov 03 07:47:53.747287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746915 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-model-cache" pod "a925815c-373f-4571-ba1a-0a0fbc93674a" (UID: "a925815c-373f-4571-ba1a-0a0fbc93674a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.747287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746946 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-home\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-home\") pod \"28861b2d-0779-4b84-bfff-76111f6068cd\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " Nov 03 07:47:53.747287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.746976 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-dshm\") pod \"28861b2d-0779-4b84-bfff-76111f6068cd\" (UID: \"28861b2d-0779-4b84-bfff-76111f6068cd\") " Nov 03 07:47:53.748904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.747763 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-tmp-dir" pod "28861b2d-0779-4b84-bfff-76111f6068cd" (UID: "28861b2d-0779-4b84-bfff-76111f6068cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.748904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.747894 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-model-cache" pod "28861b2d-0779-4b84-bfff-76111f6068cd" (UID: "28861b2d-0779-4b84-bfff-76111f6068cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.748904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.748543 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-model-cache" pod "a28a39a6-785f-4b41-91ea-af2ac05b443f" (UID: "a28a39a6-785f-4b41-91ea-af2ac05b443f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.748904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.748579 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-tmp-dir" pod "a28a39a6-785f-4b41-91ea-af2ac05b443f" (UID: "a28a39a6-785f-4b41-91ea-af2ac05b443f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.748904 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.748855 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-home" pod "28861b2d-0779-4b84-bfff-76111f6068cd" (UID: "28861b2d-0779-4b84-bfff-76111f6068cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.750852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.750823 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a28a39a6-785f-4b41-91ea-af2ac05b443f-tls-certs" pod "a28a39a6-785f-4b41-91ea-af2ac05b443f" (UID: "a28a39a6-785f-4b41-91ea-af2ac05b443f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:53.752083 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.752056 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-model-cache\") pod \"a28a39a6-785f-4b41-91ea-af2ac05b443f\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " Nov 03 07:47:53.752287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.752133 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28a39a6-785f-4b41-91ea-af2ac05b443f-kube-api-access-76tw2\" (UniqueName: \"kubernetes.io/projected/a28a39a6-785f-4b41-91ea-af2ac05b443f-kube-api-access-76tw2\") pod \"a28a39a6-785f-4b41-91ea-af2ac05b443f\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " Nov 03 07:47:53.752287 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.752167 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-tmp-dir\") pod \"a925815c-373f-4571-ba1a-0a0fbc93674a\" (UID: \"a925815c-373f-4571-ba1a-0a0fbc93674a\") " Nov 03 07:47:53.752608 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.752584 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-tmp-dir" pod "a925815c-373f-4571-ba1a-0a0fbc93674a" (UID: "a925815c-373f-4571-ba1a-0a0fbc93674a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.752830 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.752194 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-home\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-home\") pod \"a28a39a6-785f-4b41-91ea-af2ac05b443f\" (UID: \"a28a39a6-785f-4b41-91ea-af2ac05b443f\") " Nov 03 07:47:53.752905 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.752823 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-home" pod "a28a39a6-785f-4b41-91ea-af2ac05b443f" (UID: "a28a39a6-785f-4b41-91ea-af2ac05b443f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.753216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753194 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753220 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753237 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753250 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753263 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753275 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753305 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753288 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753300 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753552 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753568 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28a39a6-785f-4b41-91ea-af2ac05b443f-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.753639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753425 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28861b2d-0779-4b84-bfff-76111f6068cd-tls-certs" pod "28861b2d-0779-4b84-bfff-76111f6068cd" (UID: "28861b2d-0779-4b84-bfff-76111f6068cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:53.753639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.753451 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-dshm" pod "a925815c-373f-4571-ba1a-0a0fbc93674a" (UID: "a925815c-373f-4571-ba1a-0a0fbc93674a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.756023 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.755985 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-dshm" pod "a28a39a6-785f-4b41-91ea-af2ac05b443f" (UID: "a28a39a6-785f-4b41-91ea-af2ac05b443f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.756231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.756193 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a925815c-373f-4571-ba1a-0a0fbc93674a-tls-certs" pod "a925815c-373f-4571-ba1a-0a0fbc93674a" (UID: "a925815c-373f-4571-ba1a-0a0fbc93674a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:53.756349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.756253 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a925815c-373f-4571-ba1a-0a0fbc93674a-kube-api-access-5qpz9" pod "a925815c-373f-4571-ba1a-0a0fbc93674a" (UID: "a925815c-373f-4571-ba1a-0a0fbc93674a"). InnerVolumeSpecName "kube-api-access-5qpz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:53.756429 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.756404 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28861b2d-0779-4b84-bfff-76111f6068cd-kube-api-access-6v2lk" pod "28861b2d-0779-4b84-bfff-76111f6068cd" (UID: "28861b2d-0779-4b84-bfff-76111f6068cd"). InnerVolumeSpecName "kube-api-access-6v2lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:53.756585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.756566 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-dshm" pod "28861b2d-0779-4b84-bfff-76111f6068cd" (UID: "28861b2d-0779-4b84-bfff-76111f6068cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:53.756833 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.756807 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28a39a6-785f-4b41-91ea-af2ac05b443f-kube-api-access-76tw2" pod "a28a39a6-785f-4b41-91ea-af2ac05b443f" (UID: "a28a39a6-785f-4b41-91ea-af2ac05b443f"). InnerVolumeSpecName "kube-api-access-76tw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:53.854495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854456 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v2lk\" (UniqueName: \"kubernetes.io/projected/28861b2d-0779-4b84-bfff-76111f6068cd-kube-api-access-6v2lk\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.854495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854484 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28a39a6-785f-4b41-91ea-af2ac05b443f-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.854495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854494 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28861b2d-0779-4b84-bfff-76111f6068cd-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.854495 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854502 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76tw2\" (UniqueName: \"kubernetes.io/projected/a28a39a6-785f-4b41-91ea-af2ac05b443f-kube-api-access-76tw2\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.854877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854511 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qpz9\" (UniqueName: \"kubernetes.io/projected/a925815c-373f-4571-ba1a-0a0fbc93674a-kube-api-access-5qpz9\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.854877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854520 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a925815c-373f-4571-ba1a-0a0fbc93674a-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.854877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854529 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a925815c-373f-4571-ba1a-0a0fbc93674a-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:53.854877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:53.854536 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28861b2d-0779-4b84-bfff-76111f6068cd-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:54.178666 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.178624 4925 generic.go:356] "Generic (PLEG): container finished" podID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerID="0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349" exitCode=0 Nov 03 07:47:54.178876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.178674 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" Nov 03 07:47:54.178876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.178718 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" event={"ID":"a925815c-373f-4571-ba1a-0a0fbc93674a","Type":"ContainerDied","Data":"0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349"} Nov 03 07:47:54.178876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.178764 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk" event={"ID":"a925815c-373f-4571-ba1a-0a0fbc93674a","Type":"ContainerDied","Data":"0b0e7eb28656c83c61380fc2fd1e4c39ff0619f94eeb23849487397de1d9ff6e"} Nov 03 07:47:54.178876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.178787 4925 scope.go:122] "RemoveContainer" containerID="0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349" Nov 03 07:47:54.180411 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.180382 4925 generic.go:356] "Generic (PLEG): container finished" podID="28861b2d-0779-4b84-bfff-76111f6068cd" containerID="d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0" exitCode=0 Nov 03 07:47:54.180574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.180421 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" Nov 03 07:47:54.180574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.180461 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" event={"ID":"28861b2d-0779-4b84-bfff-76111f6068cd","Type":"ContainerDied","Data":"d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0"} Nov 03 07:47:54.180574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.180501 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s" event={"ID":"28861b2d-0779-4b84-bfff-76111f6068cd","Type":"ContainerDied","Data":"47065a3ff4dc07b91967d84247803b8c6755fb3fc566a0b198e46ae7ce22fec9"} Nov 03 07:47:54.181993 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.181970 4925 generic.go:356] "Generic (PLEG): container finished" podID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerID="0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda" exitCode=0 Nov 03 07:47:54.182129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.182032 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" Nov 03 07:47:54.182129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.182082 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" event={"ID":"a28a39a6-785f-4b41-91ea-af2ac05b443f","Type":"ContainerDied","Data":"0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda"} Nov 03 07:47:54.182263 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.182127 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8" event={"ID":"a28a39a6-785f-4b41-91ea-af2ac05b443f","Type":"ContainerDied","Data":"3db549459368ded2873ddbd6060456c04b0d95c2c60ff573876e8a9677f37ac6"} Nov 03 07:47:54.193734 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.193711 4925 scope.go:122] "RemoveContainer" containerID="0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349" Nov 03 07:47:54.194118 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:54.194087 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349\": container with ID starting with 0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349 not found: ID does not exist" containerID="0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349" Nov 03 07:47:54.194237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.194125 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349"} err="failed to get container status \"0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349\": rpc error: code = NotFound desc = could not find container \"0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349\": container with ID starting with 0d0e5be86e5feab189730d92cbde6410ba8c42bc96c30aa0b6a3e66722959349 not found: ID does not exist" Nov 03 07:47:54.194237 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.194147 4925 scope.go:122] "RemoveContainer" containerID="d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0" Nov 03 07:47:54.205004 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.204985 4925 scope.go:122] "RemoveContainer" containerID="d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0" Nov 03 07:47:54.205256 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:54.205239 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0\": container with ID starting with d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0 not found: ID does not exist" containerID="d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0" Nov 03 07:47:54.205344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.205261 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0"} err="failed to get container status \"d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0\": rpc error: code = NotFound desc = could not find container \"d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0\": container with ID starting with d5531a8c0b2f7c10ab0179ab9269197a63ce3e2274c37273f6ad7176fdfcdde0 not found: ID does not exist" Nov 03 07:47:54.205344 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.205277 4925 scope.go:122] "RemoveContainer" containerID="0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda" Nov 03 07:47:54.214297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.214281 4925 scope.go:122] "RemoveContainer" containerID="0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda" Nov 03 07:47:54.214579 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:47:54.214554 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda\": container with ID starting with 0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda not found: ID does not exist" containerID="0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda" Nov 03 07:47:54.214644 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.214588 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda"} err="failed to get container status \"0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda\": rpc error: code = NotFound desc = could not find container \"0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda\": container with ID starting with 0e2a3e71a0522932e43b32ea63e0ef307ce9f52020f1eea4967373bcb70b0bda not found: ID does not exist" Nov 03 07:47:54.349802 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.349767 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s"] Nov 03 07:47:54.359667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.359635 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/simulator-w0-6606d6-kserve-5cbb555fdc-cjn7s"] Nov 03 07:47:54.473497 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.473424 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8"] Nov 03 07:47:54.480827 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.480800 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/premium-w0-6606d6-kserve-649d9c44f8-cpns8"] Nov 03 07:47:54.595391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.595357 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk"] Nov 03 07:47:54.601384 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:54.601281 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6606d6/distinct-w0-6606d6-kserve-58d55959c-t8kbk"] Nov 03 07:47:56.033532 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:56.033492 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:56.190135 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:56.190102 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" path="/var/lib/kubelet/pods/28861b2d-0779-4b84-bfff-76111f6068cd/volumes" Nov 03 07:47:56.190527 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:56.190514 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" path="/var/lib/kubelet/pods/a28a39a6-785f-4b41-91ea-af2ac05b443f/volumes" Nov 03 07:47:56.190875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:56.190864 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" path="/var/lib/kubelet/pods/a925815c-373f-4571-ba1a-0a0fbc93674a/volumes" Nov 03 07:47:57.357907 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:57.357872 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g"] Nov 03 07:47:57.770104 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:57.770027 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:57.962254 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:57.962215 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:47:59.883631 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:47:59.883597 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:00.100896 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.100862 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:00.277713 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.277602 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" podUID="d344362f-1e23-4c7d-9829-4a989600135a" containerName="main" containerID="cri-o://6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1" gracePeriod=45 Nov 03 07:48:00.320984 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.320948 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" podUID="d344362f-1e23-4c7d-9829-4a989600135a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": dial tcp 10.132.0.118:8000: connect: connection refused" Nov 03 07:48:00.554606 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.554574 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:48:00.617266 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617238 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-tmp-dir\") pod \"d344362f-1e23-4c7d-9829-4a989600135a\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " Nov 03 07:48:00.617266 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617272 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-home\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-home\") pod \"d344362f-1e23-4c7d-9829-4a989600135a\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " Nov 03 07:48:00.617500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617331 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d344362f-1e23-4c7d-9829-4a989600135a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d344362f-1e23-4c7d-9829-4a989600135a-tls-certs\") pod \"d344362f-1e23-4c7d-9829-4a989600135a\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " Nov 03 07:48:00.617500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617353 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d344362f-1e23-4c7d-9829-4a989600135a-kube-api-access-5kkgs\" (UniqueName: \"kubernetes.io/projected/d344362f-1e23-4c7d-9829-4a989600135a-kube-api-access-5kkgs\") pod \"d344362f-1e23-4c7d-9829-4a989600135a\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " Nov 03 07:48:00.617500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617385 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-dshm\") pod \"d344362f-1e23-4c7d-9829-4a989600135a\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " Nov 03 07:48:00.617500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617436 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-model-cache\") pod \"d344362f-1e23-4c7d-9829-4a989600135a\" (UID: \"d344362f-1e23-4c7d-9829-4a989600135a\") " Nov 03 07:48:00.617718 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617524 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-tmp-dir" pod "d344362f-1e23-4c7d-9829-4a989600135a" (UID: "d344362f-1e23-4c7d-9829-4a989600135a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:00.617718 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617624 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-home" pod "d344362f-1e23-4c7d-9829-4a989600135a" (UID: "d344362f-1e23-4c7d-9829-4a989600135a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:00.617822 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617706 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-model-cache" pod "d344362f-1e23-4c7d-9829-4a989600135a" (UID: "d344362f-1e23-4c7d-9829-4a989600135a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:00.617879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617837 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.617879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617856 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.617879 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.617865 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.619705 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.619676 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-dshm" pod "d344362f-1e23-4c7d-9829-4a989600135a" (UID: "d344362f-1e23-4c7d-9829-4a989600135a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:00.619807 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.619771 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d344362f-1e23-4c7d-9829-4a989600135a-tls-certs" pod "d344362f-1e23-4c7d-9829-4a989600135a" (UID: "d344362f-1e23-4c7d-9829-4a989600135a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:00.619859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.619831 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d344362f-1e23-4c7d-9829-4a989600135a-kube-api-access-5kkgs" pod "d344362f-1e23-4c7d-9829-4a989600135a" (UID: "d344362f-1e23-4c7d-9829-4a989600135a"). InnerVolumeSpecName "kube-api-access-5kkgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:00.631858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.631815 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:00.718435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.718398 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d344362f-1e23-4c7d-9829-4a989600135a-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.718435 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.718430 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkgs\" (UniqueName: \"kubernetes.io/projected/d344362f-1e23-4c7d-9829-4a989600135a-kube-api-access-5kkgs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.718648 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:00.718446 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d344362f-1e23-4c7d-9829-4a989600135a-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:01.223174 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.223136 4925 generic.go:356] "Generic (PLEG): container finished" podID="d344362f-1e23-4c7d-9829-4a989600135a" containerID="6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1" exitCode=0 Nov 03 07:48:01.223580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.223192 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" Nov 03 07:48:01.223580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.223196 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" event={"ID":"d344362f-1e23-4c7d-9829-4a989600135a","Type":"ContainerDied","Data":"6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1"} Nov 03 07:48:01.223580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.223244 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf" event={"ID":"d344362f-1e23-4c7d-9829-4a989600135a","Type":"ContainerDied","Data":"036a5f2facedaa27c4098d679ee351f32133d3e6ef3812bca373209a4a4cecbc"} Nov 03 07:48:01.223580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.223269 4925 scope.go:122] "RemoveContainer" containerID="6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1" Nov 03 07:48:01.234183 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.234156 4925 scope.go:122] "RemoveContainer" containerID="6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1" Nov 03 07:48:01.234511 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:48:01.234483 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1\": container with ID starting with 6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1 not found: ID does not exist" containerID="6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1" Nov 03 07:48:01.234605 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.234524 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1"} err="failed to get container status \"6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1\": rpc error: code = NotFound desc = could not find container \"6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1\": container with ID starting with 6b53161a3beadec904fded729ddf4bcd74c8b17f0b124e412bbbacf829b084d1 not found: ID does not exist" Nov 03 07:48:01.393712 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.393675 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf"] Nov 03 07:48:01.444426 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:01.444396 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-eac348/test-model-eac348-kserve-57b94d7759-6jzdf"] Nov 03 07:48:02.191062 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.191031 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d344362f-1e23-4c7d-9829-4a989600135a" path="/var/lib/kubelet/pods/d344362f-1e23-4c7d-9829-4a989600135a/volumes" Nov 03 07:48:02.200658 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.200623 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:02.358754 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.358669 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" podUID="e4e875e6-0b31-447b-9069-2dd736fda882" containerName="payload-processing" containerID="cri-o://9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff" gracePeriod=25 Nov 03 07:48:02.433993 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.433963 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:02.608430 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.608408 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:48:02.623547 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.623511 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:02.735799 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.735729 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e875e6-0b31-447b-9069-2dd736fda882-kube-api-access-fm48x\" (UniqueName: \"kubernetes.io/projected/e4e875e6-0b31-447b-9069-2dd736fda882-kube-api-access-fm48x\") pod \"e4e875e6-0b31-447b-9069-2dd736fda882\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " Nov 03 07:48:02.735951 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.735826 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e875e6-0b31-447b-9069-2dd736fda882-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e875e6-0b31-447b-9069-2dd736fda882-plugins-config-volume\") pod \"e4e875e6-0b31-447b-9069-2dd736fda882\" (UID: \"e4e875e6-0b31-447b-9069-2dd736fda882\") " Nov 03 07:48:02.736285 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.736249 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e875e6-0b31-447b-9069-2dd736fda882-plugins-config-volume" pod "e4e875e6-0b31-447b-9069-2dd736fda882" (UID: "e4e875e6-0b31-447b-9069-2dd736fda882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:02.738251 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.738224 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e875e6-0b31-447b-9069-2dd736fda882-kube-api-access-fm48x" pod "e4e875e6-0b31-447b-9069-2dd736fda882" (UID: "e4e875e6-0b31-447b-9069-2dd736fda882"). InnerVolumeSpecName "kube-api-access-fm48x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:02.837494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.837461 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm48x\" (UniqueName: \"kubernetes.io/projected/e4e875e6-0b31-447b-9069-2dd736fda882-kube-api-access-fm48x\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:02.837494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:02.837488 4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e875e6-0b31-447b-9069-2dd736fda882-plugins-config-volume\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:03.130401 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.130366 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:03.235676 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.235638 4925 generic.go:356] "Generic (PLEG): container finished" podID="e4e875e6-0b31-447b-9069-2dd736fda882" containerID="9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff" exitCode=0 Nov 03 07:48:03.235877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.235691 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" Nov 03 07:48:03.235877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.235720 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" event={"ID":"e4e875e6-0b31-447b-9069-2dd736fda882","Type":"ContainerDied","Data":"9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff"} Nov 03 07:48:03.235877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.235777 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g" event={"ID":"e4e875e6-0b31-447b-9069-2dd736fda882","Type":"ContainerDied","Data":"cc768f10d0d862fddd82b5ba0b1b7a06129953c20e75f77a2b2b61af5b1ba306"} Nov 03 07:48:03.235877 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.235805 4925 scope.go:122] "RemoveContainer" containerID="9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff" Nov 03 07:48:03.245381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.245361 4925 scope.go:122] "RemoveContainer" containerID="9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff" Nov 03 07:48:03.245669 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:48:03.245642 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff\": container with ID starting with 9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff not found: ID does not exist" containerID="9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff" Nov 03 07:48:03.245729 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.245682 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff"} err="failed to get container status \"9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff\": rpc error: code = NotFound desc = could not find container \"9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff\": container with ID starting with 9ef899615edd0afd22066779642fc8f896c43cfb684c6fad4a161a76defc4dff not found: ID does not exist" Nov 03 07:48:03.406817 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.406732 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g"] Nov 03 07:48:03.457566 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:03.457532 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g"] Nov 03 07:48:04.189558 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:04.189527 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e875e6-0b31-447b-9069-2dd736fda882" path="/var/lib/kubelet/pods/e4e875e6-0b31-447b-9069-2dd736fda882/volumes" Nov 03 07:48:07.558663 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:07.558631 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:07.808538 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:07.808501 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:07.935785 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:07.935751 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:48:09.137578 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:48:09.137544 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:48:10.758779 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:10.758744 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:11.000350 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:11.000309 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:11.203468 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:11.203434 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:12.884871 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:12.884833 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:16.470899 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:16.470858 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:17.107329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.107286 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:17.515496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.515413 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl"] Nov 03 07:48:17.516268 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.516231 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" podUID="4c123d0b-147b-4710-a0fc-0022ded11038" containerName="maas-api" containerID="cri-o://931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8" gracePeriod=30 Nov 03 07:48:17.612325 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.612286 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:17.784967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.784940 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:48:17.811365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.811336 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:17.881136 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.881104 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c123d0b-147b-4710-a0fc-0022ded11038-kube-api-access-9dx7t\" (UniqueName: \"kubernetes.io/projected/4c123d0b-147b-4710-a0fc-0022ded11038-kube-api-access-9dx7t\") pod \"4c123d0b-147b-4710-a0fc-0022ded11038\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " Nov 03 07:48:17.881291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.881199 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c123d0b-147b-4710-a0fc-0022ded11038-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c123d0b-147b-4710-a0fc-0022ded11038-openshift-service-ca\") pod \"4c123d0b-147b-4710-a0fc-0022ded11038\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " Nov 03 07:48:17.881291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.881222 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls\") pod \"4c123d0b-147b-4710-a0fc-0022ded11038\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " Nov 03 07:48:17.881291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.881244 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-metrics-certs\") pod \"4c123d0b-147b-4710-a0fc-0022ded11038\" (UID: \"4c123d0b-147b-4710-a0fc-0022ded11038\") " Nov 03 07:48:17.881639 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.881606 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c123d0b-147b-4710-a0fc-0022ded11038-openshift-service-ca" pod "4c123d0b-147b-4710-a0fc-0022ded11038" (UID: "4c123d0b-147b-4710-a0fc-0022ded11038"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:17.883407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.883381 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-metrics-certs" pod "4c123d0b-147b-4710-a0fc-0022ded11038" (UID: "4c123d0b-147b-4710-a0fc-0022ded11038"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:17.883528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.883428 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls" pod "4c123d0b-147b-4710-a0fc-0022ded11038" (UID: "4c123d0b-147b-4710-a0fc-0022ded11038"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:17.883585 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.883520 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c123d0b-147b-4710-a0fc-0022ded11038-kube-api-access-9dx7t" pod "4c123d0b-147b-4710-a0fc-0022ded11038" (UID: "4c123d0b-147b-4710-a0fc-0022ded11038"). InnerVolumeSpecName "kube-api-access-9dx7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:17.982286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.982250 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dx7t\" (UniqueName: \"kubernetes.io/projected/4c123d0b-147b-4710-a0fc-0022ded11038-kube-api-access-9dx7t\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:17.982286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.982275 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c123d0b-147b-4710-a0fc-0022ded11038-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:17.982286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.982286 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:17.982286 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:17.982296 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c123d0b-147b-4710-a0fc-0022ded11038-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:18.135291 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.135252 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:18.323794 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.323754 4925 generic.go:356] "Generic (PLEG): container finished" podID="4c123d0b-147b-4710-a0fc-0022ded11038" containerID="931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8" exitCode=0 Nov 03 07:48:18.323989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.323802 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" Nov 03 07:48:18.323989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.323848 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" event={"ID":"4c123d0b-147b-4710-a0fc-0022ded11038","Type":"ContainerDied","Data":"931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8"} Nov 03 07:48:18.323989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.323917 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl" event={"ID":"4c123d0b-147b-4710-a0fc-0022ded11038","Type":"ContainerDied","Data":"e2d069b4929e3076615850e99ba91735d6cdc37648209c39095ec6c475fa67d6"} Nov 03 07:48:18.323989 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.323948 4925 scope.go:122] "RemoveContainer" containerID="931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8" Nov 03 07:48:18.333331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.333310 4925 scope.go:122] "RemoveContainer" containerID="931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8" Nov 03 07:48:18.333619 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:48:18.333597 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8\": container with ID starting with 931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8 not found: ID does not exist" containerID="931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8" Nov 03 07:48:18.333701 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.333632 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8"} err="failed to get container status \"931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8\": rpc error: code = NotFound desc = could not find container \"931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8\": container with ID starting with 931c1bb6ff3b35226d74f2a569d78b744e8771cdae766dee3b1eb497266dfbe8 not found: ID does not exist" Nov 03 07:48:18.498193 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.498112 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl"] Nov 03 07:48:18.548852 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:18.548817 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6606d6-ff9d7565-fg2sl"] Nov 03 07:48:19.269559 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:19.269515 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:19.552967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:19.552879 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:19.757716 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:19.757679 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:20.189462 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:20.189426 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c123d0b-147b-4710-a0fc-0022ded11038" path="/var/lib/kubelet/pods/4c123d0b-147b-4710-a0fc-0022ded11038/volumes" Nov 03 07:48:20.212411 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:20.212374 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:23.877556 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:23.876239 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:24.073194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:24.073163 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:24.263178 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:24.263081 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:25.440328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:25.440295 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:25.652494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:25.652446 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:26.036460 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:26.036422 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:27.415686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:27.415651 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:27.607031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:27.606961 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:27.793737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:27.793643 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:28.118095 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:28.118063 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:48:28.179764 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:28.179714 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:28.470345 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:28.470252 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:29.084664 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:29.084621 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:31.223739 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:31.223698 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:36.338971 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:36.338929 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:36.535488 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:36.535441 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:36.726515 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:36.726419 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:37.173189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.173154 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:37.652749 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.652707 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9"] Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653117 4925 cpu_manager.go:414] "containerMap: removing container" podUID="ac12005a-d91c-404f-ab8c-5c94f723b2fc" containerName="main" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653129 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac12005a-d91c-404f-ab8c-5c94f723b2fc" containerName="main" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653138 4925 cpu_manager.go:414] "containerMap: removing container" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653144 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653153 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e4e875e6-0b31-447b-9069-2dd736fda882" containerName="payload-processing" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653159 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e875e6-0b31-447b-9069-2dd736fda882" containerName="payload-processing" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653165 4925 cpu_manager.go:414] "containerMap: removing container" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653170 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653183 4925 cpu_manager.go:414] "containerMap: removing container" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.653184 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653189 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653196 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f74065d0-b412-4763-8da5-9ec45c612370" containerName="maas-api" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653201 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74065d0-b412-4763-8da5-9ec45c612370" containerName="maas-api" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653206 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653211 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653217 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d344362f-1e23-4c7d-9829-4a989600135a" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653222 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d344362f-1e23-4c7d-9829-4a989600135a" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653230 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653234 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653240 4925 cpu_manager.go:414] "containerMap: removing container" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653245 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653252 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653256 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653262 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e2a3483b-f650-46e3-a927-002cd5cf94ca" containerName="maas-api" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653266 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a3483b-f650-46e3-a927-002cd5cf94ca" containerName="maas-api" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653272 4925 cpu_manager.go:414] "containerMap: removing container" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653277 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653284 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653287 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653293 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653297 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653302 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653306 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653313 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653317 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653322 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653325 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653334 4925 cpu_manager.go:414] "containerMap: removing container" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653338 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653343 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653348 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653354 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653358 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653366 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653370 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653379 4925 cpu_manager.go:414] "containerMap: removing container" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653383 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653389 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653393 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653402 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653407 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653418 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4c123d0b-147b-4710-a0fc-0022ded11038" containerName="maas-api" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653422 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c123d0b-147b-4710-a0fc-0022ded11038" containerName="maas-api" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653434 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653438 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653444 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4506a9e5-97e5-48d6-b3bb-20ca29482c37" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653448 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4506a9e5-97e5-48d6-b3bb-20ca29482c37" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653504 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28a39a6-785f-4b41-91ea-af2ac05b443f" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653512 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6802fb28-f1af-4a5a-b353-eb333060b469" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653520 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653525 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653530 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653536 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653542 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653548 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f65cfe2-7f4e-40d9-9fec-96b82223194c" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653554 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7756b04-4fca-44d7-af02-0bef47d8ae3b" containerName="cleanup" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653561 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="28861b2d-0779-4b84-bfff-76111f6068cd" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653566 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a925815c-373f-4571-ba1a-0a0fbc93674a" containerName="main" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653572 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a571841e-f2f4-414a-a6b4-cb3d40f0ad29" containerName="istio-proxy" Nov 03 07:48:37.653552 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653578 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653584 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c123d0b-147b-4710-a0fc-0022ded11038" containerName="maas-api" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653589 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653595 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f264c5c8-6531-4981-bafe-61609a4a60fa" containerName="istio-proxy" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653600 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4506a9e5-97e5-48d6-b3bb-20ca29482c37" containerName="main" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653605 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac12005a-d91c-404f-ab8c-5c94f723b2fc" containerName="main" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653610 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e875e6-0b31-447b-9069-2dd736fda882" containerName="payload-processing" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653615 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653619 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74065d0-b412-4763-8da5-9ec45c612370" containerName="maas-api" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653624 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d344362f-1e23-4c7d-9829-4a989600135a" containerName="main" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653628 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52ae06b-ed08-47b0-be5f-2e522283b659" containerName="istio-proxy" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653632 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653637 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc1842f-4ade-4a0b-9bc0-af7fabc0c92e" containerName="main" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653641 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a3483b-f650-46e3-a927-002cd5cf94ca" containerName="maas-api" Nov 03 07:48:37.655465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.653646 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="28910ce8-0366-4962-94a8-4ed5667820a3" containerName="main" Nov 03 07:48:37.656759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.656739 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.716476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.716444 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9"] Nov 03 07:48:37.743099 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.743069 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtlz\" (UniqueName: \"kubernetes.io/projected/4c74c14f-31d4-49b1-ab64-8aa866ef9791-kube-api-access-lqtlz\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.743247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.743189 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.743247 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.743221 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.764298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.764268 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 07:48:37.764298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.764269 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 07:48:37.764298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.764269 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 07:48:37.764484 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.764320 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:48:37.844594 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.844566 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.844727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.844603 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.844727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.844646 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtlz\" (UniqueName: \"kubernetes.io/projected/4c74c14f-31d4-49b1-ab64-8aa866ef9791-kube-api-access-lqtlz\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.845349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.845330 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.847139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.847117 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.909244 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.909175 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtlz\" (UniqueName: \"kubernetes.io/projected/4c74c14f-31d4-49b1-ab64-8aa866ef9791-kube-api-access-lqtlz\") pod \"maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:37.969496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:37.969466 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:38.351691 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:38.351663 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9"] Nov 03 07:48:38.354406 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:48:38.354374 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c74c14f_31d4_49b1_ab64_8aa866ef9791.slice/crio-4eeeace612df3286f7defde987272713e0b1f48ac98a05db099c5455860f54d1 WatchSource:0}: Error finding container 4eeeace612df3286f7defde987272713e0b1f48ac98a05db099c5455860f54d1: Status 404 returned error can't find the container with id 4eeeace612df3286f7defde987272713e0b1f48ac98a05db099c5455860f54d1 Nov 03 07:48:38.438392 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:38.438359 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" event={"ID":"4c74c14f-31d4-49b1-ab64-8aa866ef9791","Type":"ContainerStarted","Data":"4eeeace612df3286f7defde987272713e0b1f48ac98a05db099c5455860f54d1"} Nov 03 07:48:39.137670 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:48:39.137627 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:48:39.445082 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:39.444986 4925 generic.go:356] "Generic (PLEG): container finished" podID="4c74c14f-31d4-49b1-ab64-8aa866ef9791" containerID="7f050cee72155ed3e3615cb922b0ce60fcc86ac3e68ea7bf829df0fadea588db" exitCode=0 Nov 03 07:48:39.445234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:39.445073 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" event={"ID":"4c74c14f-31d4-49b1-ab64-8aa866ef9791","Type":"ContainerDied","Data":"7f050cee72155ed3e3615cb922b0ce60fcc86ac3e68ea7bf829df0fadea588db"} Nov 03 07:48:40.582331 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.582302 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:40.669531 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.669501 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-xks-ca\") pod \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " Nov 03 07:48:40.669687 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.669567 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-openshift-ca\") pod \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " Nov 03 07:48:40.669687 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.669643 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c74c14f-31d4-49b1-ab64-8aa866ef9791-kube-api-access-lqtlz\" (UniqueName: \"kubernetes.io/projected/4c74c14f-31d4-49b1-ab64-8aa866ef9791-kube-api-access-lqtlz\") pod \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\" (UID: \"4c74c14f-31d4-49b1-ab64-8aa866ef9791\") " Nov 03 07:48:40.669980 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.669947 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-openshift-ca" pod "4c74c14f-31d4-49b1-ab64-8aa866ef9791" (UID: "4c74c14f-31d4-49b1-ab64-8aa866ef9791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:40.671543 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.671518 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-xks-ca" pod "4c74c14f-31d4-49b1-ab64-8aa866ef9791" (UID: "4c74c14f-31d4-49b1-ab64-8aa866ef9791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:40.671669 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.671654 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c74c14f-31d4-49b1-ab64-8aa866ef9791-kube-api-access-lqtlz" pod "4c74c14f-31d4-49b1-ab64-8aa866ef9791" (UID: "4c74c14f-31d4-49b1-ab64-8aa866ef9791"). InnerVolumeSpecName "kube-api-access-lqtlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:40.770315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.770245 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-xks-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:40.770315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.770272 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c74c14f-31d4-49b1-ab64-8aa866ef9791-maas-api-openshift-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:40.770315 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:40.770282 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtlz\" (UniqueName: \"kubernetes.io/projected/4c74c14f-31d4-49b1-ab64-8aa866ef9791-kube-api-access-lqtlz\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.456208 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:41.456173 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" Nov 03 07:48:41.456363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:41.456207 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9" event={"ID":"4c74c14f-31d4-49b1-ab64-8aa866ef9791","Type":"ContainerDied","Data":"4eeeace612df3286f7defde987272713e0b1f48ac98a05db099c5455860f54d1"} Nov 03 07:48:41.456363 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:41.456259 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eeeace612df3286f7defde987272713e0b1f48ac98a05db099c5455860f54d1" Nov 03 07:48:41.780798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:41.780714 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:42.087551 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:42.087511 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn"] Nov 03 07:48:42.136839 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:42.136802 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6"] Nov 03 07:48:42.238310 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:42.238274 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:46.940031 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:46.939976 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:49.017509 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:49.017418 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:49.207642 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:49.207609 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:51.934202 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:51.934173 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:48:56.392957 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.392890 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72"] Nov 03 07:48:56.393485 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393458 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4c74c14f-31d4-49b1-ab64-8aa866ef9791" containerName="revoke-keys" Nov 03 07:48:56.393654 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393635 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c74c14f-31d4-49b1-ab64-8aa866ef9791" containerName="revoke-keys" Nov 03 07:48:56.393730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393670 4925 cpu_manager.go:414] "containerMap: removing container" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:56.393730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393676 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:56.393730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393686 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:56.393730 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393691 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:56.393875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393771 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6687c623-2897-4891-aefe-b9d285910a84" containerName="cleanup" Nov 03 07:48:56.393875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393781 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="38515a4d-17c9-45dc-995a-dea48bd883dc" containerName="cleanup" Nov 03 07:48:56.393875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393790 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e52396-ef05-4008-919f-1610ce4a4aab" containerName="cleanup" Nov 03 07:48:56.393875 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.393803 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c74c14f-31d4-49b1-ab64-8aa866ef9791" containerName="revoke-keys" Nov 03 07:48:56.397123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.397103 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.455689 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.455652 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72"] Nov 03 07:48:56.503947 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.503912 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 07:48:56.503947 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.503942 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 07:48:56.504172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.503931 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 07:48:56.504172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.503993 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:48:56.505275 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.505251 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.505415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.505297 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gjb\" (UniqueName: \"kubernetes.io/projected/427024d8-76f8-4759-8018-3deb63b09fbc-kube-api-access-q5gjb\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.505415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.505317 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.606786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.606748 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gjb\" (UniqueName: \"kubernetes.io/projected/427024d8-76f8-4759-8018-3deb63b09fbc-kube-api-access-q5gjb\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.606786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.606786 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.607083 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.606860 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.607656 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.607594 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.609451 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.609429 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.671338 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.671272 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gjb\" (UniqueName: \"kubernetes.io/projected/427024d8-76f8-4759-8018-3deb63b09fbc-kube-api-access-q5gjb\") pod \"maas-api-revoke-sub-73e423dd-fjb72\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:56.709737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:56.709708 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:57.088498 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.088474 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72"] Nov 03 07:48:57.090186 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:48:57.090161 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427024d8_76f8_4759_8018_3deb63b09fbc.slice/crio-569d3596fdbf7d38c86172cda182cfbe79c96a200cb513efff9b39b61e16313f WatchSource:0}: Error finding container 569d3596fdbf7d38c86172cda182cfbe79c96a200cb513efff9b39b61e16313f: Status 404 returned error can't find the container with id 569d3596fdbf7d38c86172cda182cfbe79c96a200cb513efff9b39b61e16313f Nov 03 07:48:57.108859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.108834 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" containerID="cri-o://2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a" gracePeriod=45 Nov 03 07:48:57.156629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.156596 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" containerID="cri-o://5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c" gracePeriod=45 Nov 03 07:48:57.272142 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.272106 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": dial tcp 10.132.0.55:8000: connect: connection refused" Nov 03 07:48:57.389391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.389367 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:48:57.424468 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.424444 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:48:57.515373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515295 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-kube-api-access-zf9n4\" (UniqueName: \"kubernetes.io/projected/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-kube-api-access-zf9n4\") pod \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " Nov 03 07:48:57.515373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515350 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-kube-api-access-xkf5w\" (UniqueName: \"kubernetes.io/projected/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-kube-api-access-xkf5w\") pod \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " Nov 03 07:48:57.515373 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515372 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-home\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-home\") pod \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " Nov 03 07:48:57.515636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515476 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tls-certs\") pod \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " Nov 03 07:48:57.515636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515522 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tls-certs\") pod \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " Nov 03 07:48:57.515636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515599 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-home" pod "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" (UID: "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.515796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515642 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tmp-dir\") pod \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " Nov 03 07:48:57.515796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515681 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tmp-dir\") pod \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " Nov 03 07:48:57.515796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515713 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-model-cache\") pod \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " Nov 03 07:48:57.515796 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515770 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-home\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-home\") pod \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " Nov 03 07:48:57.515997 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515821 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-dshm\") pod \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " Nov 03 07:48:57.515997 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515857 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-dshm\") pod \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\" (UID: \"f80aa4db-3d6d-4f15-ab7c-09587d3285b4\") " Nov 03 07:48:57.515997 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.515893 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-model-cache\") pod \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\" (UID: \"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e\") " Nov 03 07:48:57.516220 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.516060 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tmp-dir" pod "f80aa4db-3d6d-4f15-ab7c-09587d3285b4" (UID: "f80aa4db-3d6d-4f15-ab7c-09587d3285b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.516285 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.516267 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-model-cache" pod "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" (UID: "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.516449 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.516430 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-model-cache" pod "f80aa4db-3d6d-4f15-ab7c-09587d3285b4" (UID: "f80aa4db-3d6d-4f15-ab7c-09587d3285b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.516610 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.516592 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-home" pod "f80aa4db-3d6d-4f15-ab7c-09587d3285b4" (UID: "f80aa4db-3d6d-4f15-ab7c-09587d3285b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.516783 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tmp-dir" pod "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" (UID: "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.516833 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.516852 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.518980 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-kube-api-access-xkf5w" pod "f80aa4db-3d6d-4f15-ab7c-09587d3285b4" (UID: "f80aa4db-3d6d-4f15-ab7c-09587d3285b4"). InnerVolumeSpecName "kube-api-access-xkf5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.519223 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-dshm" pod "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" (UID: "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.519414 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-kube-api-access-zf9n4" pod "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" (UID: "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e"). InnerVolumeSpecName "kube-api-access-zf9n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.520045 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tls-certs" pod "f80aa4db-3d6d-4f15-ab7c-09587d3285b4" (UID: "f80aa4db-3d6d-4f15-ab7c-09587d3285b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.520113 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tls-certs" pod "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" (UID: "b9b0ae12-0d99-4e6b-ab78-87ae0287b67e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:57.523044 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.520715 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-dshm" pod "f80aa4db-3d6d-4f15-ab7c-09587d3285b4" (UID: "f80aa4db-3d6d-4f15-ab7c-09587d3285b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:57.549887 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.549853 4925 generic.go:356] "Generic (PLEG): container finished" podID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerID="5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c" exitCode=0 Nov 03 07:48:57.550081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.549891 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" Nov 03 07:48:57.550081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.549938 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" event={"ID":"f80aa4db-3d6d-4f15-ab7c-09587d3285b4","Type":"ContainerDied","Data":"5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c"} Nov 03 07:48:57.550081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.549991 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6" event={"ID":"f80aa4db-3d6d-4f15-ab7c-09587d3285b4","Type":"ContainerDied","Data":"1d1e60da13a1b67ced439a883be3b08b89559cd3fa000362fcafd421656ee623"} Nov 03 07:48:57.550081 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.550031 4925 scope.go:122] "RemoveContainer" containerID="5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c" Nov 03 07:48:57.551558 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.551537 4925 generic.go:356] "Generic (PLEG): container finished" podID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerID="2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a" exitCode=0 Nov 03 07:48:57.551676 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.551659 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" event={"ID":"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e","Type":"ContainerDied","Data":"2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a"} Nov 03 07:48:57.551723 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.551693 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" event={"ID":"b9b0ae12-0d99-4e6b-ab78-87ae0287b67e","Type":"ContainerDied","Data":"c4bd447affaaa609965b420c2138758108da0e9cd126cb166d0725077e885993"} Nov 03 07:48:57.551774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.551762 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn" Nov 03 07:48:57.569048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.567747 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" event={"ID":"427024d8-76f8-4759-8018-3deb63b09fbc","Type":"ContainerStarted","Data":"baab99d50e7723ae08b67ab6c07d1703b380245da26028e7a177d0de805ad350"} Nov 03 07:48:57.569048 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.567841 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" event={"ID":"427024d8-76f8-4759-8018-3deb63b09fbc","Type":"ContainerStarted","Data":"569d3596fdbf7d38c86172cda182cfbe79c96a200cb513efff9b39b61e16313f"} Nov 03 07:48:57.570144 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.569972 4925 scope.go:122] "RemoveContainer" containerID="5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c" Nov 03 07:48:57.570704 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:48:57.570679 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c\": container with ID starting with 5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c not found: ID does not exist" containerID="5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c" Nov 03 07:48:57.570803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.570714 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c"} err="failed to get container status \"5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c\": rpc error: code = NotFound desc = could not find container \"5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c\": container with ID starting with 5a3df7341060bc22296e4d4e9b86ac1b39f1dff1aa4b67dbd773d1af0d526b9c not found: ID does not exist" Nov 03 07:48:57.570803 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.570736 4925 scope.go:122] "RemoveContainer" containerID="2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a" Nov 03 07:48:57.580233 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.580213 4925 scope.go:122] "RemoveContainer" containerID="2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a" Nov 03 07:48:57.580486 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:48:57.580466 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a\": container with ID starting with 2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a not found: ID does not exist" containerID="2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a" Nov 03 07:48:57.580572 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.580489 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a"} err="failed to get container status \"2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a\": rpc error: code = NotFound desc = could not find container \"2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a\": container with ID starting with 2a886e1502973ab21869a83d4b7dbd2f1e39b39d19d4495f85ec764c5b99b90a not found: ID does not exist" Nov 03 07:48:57.618298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618263 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618298 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf9n4\" (UniqueName: \"kubernetes.io/projected/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-kube-api-access-zf9n4\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618314 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkf5w\" (UniqueName: \"kubernetes.io/projected/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-kube-api-access-xkf5w\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618329 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618343 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618356 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618369 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618381 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618392 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.618491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.618404 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80aa4db-3d6d-4f15-ab7c-09587d3285b4-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:57.756900 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.756849 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" podStartSLOduration=1.756833736 podStartE2EDuration="1.756833736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:48:57.754975553 +0000 UTC m=+2997.688504211" watchObservedRunningTime="2026-11-03 07:48:57.756833736 +0000 UTC m=+2997.690362370" Nov 03 07:48:57.873831 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.873795 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn"] Nov 03 07:48:57.878876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.878844 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/distinct-w4-ffee69-kserve-7ddf87c8c8-hz4pn"] Nov 03 07:48:57.993124 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.993091 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6"] Nov 03 07:48:57.999391 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:57.999365 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ffee69/simulator-w4-ffee69-kserve-7cbbd5dc86-l25f6"] Nov 03 07:48:58.189652 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:58.189579 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" path="/var/lib/kubelet/pods/b9b0ae12-0d99-4e6b-ab78-87ae0287b67e/volumes" Nov 03 07:48:58.190056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:58.190032 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" path="/var/lib/kubelet/pods/f80aa4db-3d6d-4f15-ab7c-09587d3285b4/volumes" Nov 03 07:48:58.578750 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:58.578717 4925 generic.go:356] "Generic (PLEG): container finished" podID="427024d8-76f8-4759-8018-3deb63b09fbc" containerID="baab99d50e7723ae08b67ab6c07d1703b380245da26028e7a177d0de805ad350" exitCode=0 Nov 03 07:48:58.579158 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:58.578772 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" event={"ID":"427024d8-76f8-4759-8018-3deb63b09fbc","Type":"ContainerDied","Data":"baab99d50e7723ae08b67ab6c07d1703b380245da26028e7a177d0de805ad350"} Nov 03 07:48:59.726742 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.726715 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:48:59.836651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.836563 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-openshift-ca\") pod \"427024d8-76f8-4759-8018-3deb63b09fbc\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " Nov 03 07:48:59.836651 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.836608 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-xks-ca\") pod \"427024d8-76f8-4759-8018-3deb63b09fbc\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " Nov 03 07:48:59.836870 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.836668 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427024d8-76f8-4759-8018-3deb63b09fbc-kube-api-access-q5gjb\" (UniqueName: \"kubernetes.io/projected/427024d8-76f8-4759-8018-3deb63b09fbc-kube-api-access-q5gjb\") pod \"427024d8-76f8-4759-8018-3deb63b09fbc\" (UID: \"427024d8-76f8-4759-8018-3deb63b09fbc\") " Nov 03 07:48:59.837074 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.837047 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-openshift-ca" pod "427024d8-76f8-4759-8018-3deb63b09fbc" (UID: "427024d8-76f8-4759-8018-3deb63b09fbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:59.838912 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.838889 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-xks-ca" pod "427024d8-76f8-4759-8018-3deb63b09fbc" (UID: "427024d8-76f8-4759-8018-3deb63b09fbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:59.839071 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.839051 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427024d8-76f8-4759-8018-3deb63b09fbc-kube-api-access-q5gjb" pod "427024d8-76f8-4759-8018-3deb63b09fbc" (UID: "427024d8-76f8-4759-8018-3deb63b09fbc"). InnerVolumeSpecName "kube-api-access-q5gjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:59.937710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.937677 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gjb\" (UniqueName: \"kubernetes.io/projected/427024d8-76f8-4759-8018-3deb63b09fbc-kube-api-access-q5gjb\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:59.937710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.937706 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-openshift-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:59.937710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:48:59.937716 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427024d8-76f8-4759-8018-3deb63b09fbc-maas-api-xks-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:00.590155 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:00.590127 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" Nov 03 07:49:00.590329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:00.590163 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72" event={"ID":"427024d8-76f8-4759-8018-3deb63b09fbc","Type":"ContainerDied","Data":"569d3596fdbf7d38c86172cda182cfbe79c96a200cb513efff9b39b61e16313f"} Nov 03 07:49:00.590329 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:00.590208 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="569d3596fdbf7d38c86172cda182cfbe79c96a200cb513efff9b39b61e16313f" Nov 03 07:49:01.890759 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:01.890726 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:49:01.894208 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:01.894178 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:49:01.898334 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:01.898313 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:49:01.902093 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:01.902077 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:49:07.910126 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:07.910089 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:09.137952 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:49:09.137913 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:49:11.832402 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:11.832311 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:13.301189 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:13.301146 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:49:13.417302 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:13.417267 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:13.723806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:13.723768 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:13.929025 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:13.928982 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9"] Nov 03 07:49:13.979885 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:13.979822 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ffee69-w75j9"] Nov 03 07:49:14.189343 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:14.189309 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c74c14f-31d4-49b1-ab64-8aa866ef9791" path="/var/lib/kubelet/pods/4c74c14f-31d4-49b1-ab64-8aa866ef9791/volumes" Nov 03 07:49:14.378423 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:14.378389 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:19.722677 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:19.722639 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:19.916442 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:19.916396 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:22.252964 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:22.252930 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:22.730255 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:22.730215 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:22.919424 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:22.919392 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:23.115271 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:23.115227 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:27.068211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:27.068176 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:49:28.238781 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.238749 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:49:28.465211 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.465166 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-xpwnt"] Nov 03 07:49:28.465422 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.465375 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" podUID="d393385c-b146-42fa-8963-21439be22464" containerName="manager" containerID="cri-o://913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc" gracePeriod=30 Nov 03 07:49:28.723965 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.723936 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:49:28.742173 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.742142 4925 generic.go:356] "Generic (PLEG): container finished" podID="d393385c-b146-42fa-8963-21439be22464" containerID="913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc" exitCode=0 Nov 03 07:49:28.742327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.742189 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" Nov 03 07:49:28.742327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.742225 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" event={"ID":"d393385c-b146-42fa-8963-21439be22464","Type":"ContainerDied","Data":"913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc"} Nov 03 07:49:28.742327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.742281 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-xpwnt" event={"ID":"d393385c-b146-42fa-8963-21439be22464","Type":"ContainerDied","Data":"d55c93a78d3b1155a8ad30647dddb5a18300c35097ee847446eeaaa6ced42b3a"} Nov 03 07:49:28.742327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.742303 4925 scope.go:122] "RemoveContainer" containerID="913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc" Nov 03 07:49:28.753407 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.753383 4925 scope.go:122] "RemoveContainer" containerID="913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc" Nov 03 07:49:28.753686 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:49:28.753660 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc\": container with ID starting with 913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc not found: ID does not exist" containerID="913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc" Nov 03 07:49:28.753773 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.753694 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc"} err="failed to get container status \"913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc\": rpc error: code = NotFound desc = could not find container \"913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc\": container with ID starting with 913322b2e7f59d8438f58ff220d043e46d98dee6d66231888255d8edbf3ddafc not found: ID does not exist" Nov 03 07:49:28.785804 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.785768 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-webhook-cert\") pod \"d393385c-b146-42fa-8963-21439be22464\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " Nov 03 07:49:28.785973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.785859 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d393385c-b146-42fa-8963-21439be22464-kube-api-access-rmwp5\" (UniqueName: \"kubernetes.io/projected/d393385c-b146-42fa-8963-21439be22464-kube-api-access-rmwp5\") pod \"d393385c-b146-42fa-8963-21439be22464\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " Nov 03 07:49:28.785973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.785906 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-metrics-certs\") pod \"d393385c-b146-42fa-8963-21439be22464\" (UID: \"d393385c-b146-42fa-8963-21439be22464\") " Nov 03 07:49:28.788199 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.788171 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d393385c-b146-42fa-8963-21439be22464-kube-api-access-rmwp5" pod "d393385c-b146-42fa-8963-21439be22464" (UID: "d393385c-b146-42fa-8963-21439be22464"). InnerVolumeSpecName "kube-api-access-rmwp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:49:28.788596 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.788569 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-webhook-cert" pod "d393385c-b146-42fa-8963-21439be22464" (UID: "d393385c-b146-42fa-8963-21439be22464"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:49:28.788693 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.788674 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-metrics-certs" pod "d393385c-b146-42fa-8963-21439be22464" (UID: "d393385c-b146-42fa-8963-21439be22464"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:49:28.886920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.886883 4925 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-webhook-cert\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.886920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.886915 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmwp5\" (UniqueName: \"kubernetes.io/projected/d393385c-b146-42fa-8963-21439be22464-kube-api-access-rmwp5\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.886920 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:28.886928 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393385c-b146-42fa-8963-21439be22464-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:29.170032 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:29.169928 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-xpwnt"] Nov 03 07:49:29.177892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:29.177865 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-xpwnt"] Nov 03 07:49:30.189632 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:30.189594 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d393385c-b146-42fa-8963-21439be22464" path="/var/lib/kubelet/pods/d393385c-b146-42fa-8963-21439be22464/volumes" Nov 03 07:49:39.136610 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:49:39.136571 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:49:44.255244 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255203 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz"] Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255743 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255759 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255777 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255782 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255793 4925 cpu_manager.go:414] "containerMap: removing container" podUID="427024d8-76f8-4759-8018-3deb63b09fbc" containerName="revoke-keys" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255797 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427024d8-76f8-4759-8018-3deb63b09fbc" containerName="revoke-keys" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255812 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d393385c-b146-42fa-8963-21439be22464" containerName="manager" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255817 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d393385c-b146-42fa-8963-21439be22464" containerName="manager" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255873 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d393385c-b146-42fa-8963-21439be22464" containerName="manager" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255883 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80aa4db-3d6d-4f15-ab7c-09587d3285b4" containerName="main" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255891 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b0ae12-0d99-4e6b-ab78-87ae0287b67e" containerName="main" Nov 03 07:49:44.257675 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.255899 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="427024d8-76f8-4759-8018-3deb63b09fbc" containerName="revoke-keys" Nov 03 07:49:44.258860 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.258840 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.305786 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.305754 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52"] Nov 03 07:49:44.309206 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.309186 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.323091 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.323074 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89ggc\"" Nov 03 07:49:44.330822 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.330799 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b23567-bced-4d00-977f-73e8ccedce7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.330925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.330828 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.330925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.330848 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvl2\" (UniqueName: \"kubernetes.io/projected/03b23567-bced-4d00-977f-73e8ccedce7c-kube-api-access-sgvl2\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.330925 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.330871 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b23567-bced-4d00-977f-73e8ccedce7c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.331078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.330986 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b23567-bced-4d00-977f-73e8ccedce7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.331078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.331033 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.331078 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.331067 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.331231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.331135 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.331231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.331193 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.364571 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.364543 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz"] Nov 03 07:49:44.421906 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.421869 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52"] Nov 03 07:49:44.432591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432565 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.432591 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432599 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.432813 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432627 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.432813 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432710 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.432813 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432754 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.432813 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432784 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b23567-bced-4d00-977f-73e8ccedce7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433005 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432912 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433005 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432961 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvl2\" (UniqueName: \"kubernetes.io/projected/03b23567-bced-4d00-977f-73e8ccedce7c-kube-api-access-sgvl2\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433005 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.432993 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.433193 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433064 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433193 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433113 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b23567-bced-4d00-977f-73e8ccedce7c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433193 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433175 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.433364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433208 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.433364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433241 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b23567-bced-4d00-977f-73e8ccedce7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433271 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433281 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433312 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.433364 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433338 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.433680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433410 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433466 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbhg\" (UniqueName: \"kubernetes.io/projected/2ff1a6c9-e407-4541-a669-ad0f433d25b0-kube-api-access-nsbhg\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.433680 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433669 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433702 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b23567-bced-4d00-977f-73e8ccedce7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.433792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.433730 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.435414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.435383 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b23567-bced-4d00-977f-73e8ccedce7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.436185 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.436166 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b23567-bced-4d00-977f-73e8ccedce7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.497600 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.497564 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b23567-bced-4d00-977f-73e8ccedce7c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.497732 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.497711 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvl2\" (UniqueName: \"kubernetes.io/projected/03b23567-bced-4d00-977f-73e8ccedce7c-kube-api-access-sgvl2\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jgbz\" (UID: \"03b23567-bced-4d00-977f-73e8ccedce7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.534475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534402 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.534475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534435 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.534475 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534460 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.534783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534588 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.534783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534624 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.534783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534655 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.534783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534681 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.534783 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534742 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbhg\" (UniqueName: \"kubernetes.io/projected/2ff1a6c9-e407-4541-a669-ad0f433d25b0-kube-api-access-nsbhg\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.535088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534795 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.535088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534846 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.535088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534916 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.535088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.534958 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.535088 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.535075 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.535496 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.535478 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.537132 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.537110 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.537297 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.537280 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.572207 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.572171 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:44.598955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.598919 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff1a6c9-e407-4541-a669-ad0f433d25b0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.599109 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.598995 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbhg\" (UniqueName: \"kubernetes.io/projected/2ff1a6c9-e407-4541-a669-ad0f433d25b0-kube-api-access-nsbhg\") pod \"maas-default-gateway-openshift-default-58d898bd55-xqv52\" (UID: \"2ff1a6c9-e407-4541-a669-ad0f433d25b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:44.622065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:44.622035 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:45.001477 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.001427 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz"] Nov 03 07:49:45.038895 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:49:45.038857 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b23567_bced_4d00_977f_73e8ccedce7c.slice/crio-b33d96d512b082e974d1666eb2876424c8b5a108aa392b3e69da4ef8db4c4b83 WatchSource:0}: Error finding container b33d96d512b082e974d1666eb2876424c8b5a108aa392b3e69da4ef8db4c4b83: Status 404 returned error can't find the container with id b33d96d512b082e974d1666eb2876424c8b5a108aa392b3e69da4ef8db4c4b83 Nov 03 07:49:45.040946 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.040914 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:49:45.041036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.040984 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:49:45.041094 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.041033 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:49:45.055894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.055872 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52"] Nov 03 07:49:45.057134 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:49:45.057108 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff1a6c9_e407_4541_a669_ad0f433d25b0.slice/crio-5e8f2b11aa0d72b31fcf0145347dbb59edea7bd77b2d806cf9920e9091143f29 WatchSource:0}: Error finding container 5e8f2b11aa0d72b31fcf0145347dbb59edea7bd77b2d806cf9920e9091143f29: Status 404 returned error can't find the container with id 5e8f2b11aa0d72b31fcf0145347dbb59edea7bd77b2d806cf9920e9091143f29 Nov 03 07:49:45.059511 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.059482 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:49:45.059590 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.059543 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:49:45.059590 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.059574 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:49:45.831219 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.831184 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" event={"ID":"2ff1a6c9-e407-4541-a669-ad0f433d25b0","Type":"ContainerStarted","Data":"99b6d845fe84eddf3c61eae118bcd219dee75f4f72810865066f954523996e6d"} Nov 03 07:49:45.831707 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.831683 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" event={"ID":"2ff1a6c9-e407-4541-a669-ad0f433d25b0","Type":"ContainerStarted","Data":"5e8f2b11aa0d72b31fcf0145347dbb59edea7bd77b2d806cf9920e9091143f29"} Nov 03 07:49:45.832878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.832859 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" event={"ID":"03b23567-bced-4d00-977f-73e8ccedce7c","Type":"ContainerStarted","Data":"61036a14cbbbf2cc5e07dc9619840cfb5d5ad1e44b3df02a65917cb6f1c19b9a"} Nov 03 07:49:45.832956 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.832894 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" event={"ID":"03b23567-bced-4d00-977f-73e8ccedce7c","Type":"ContainerStarted","Data":"b33d96d512b082e974d1666eb2876424c8b5a108aa392b3e69da4ef8db4c4b83"} Nov 03 07:49:45.977735 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:45.977667 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podStartSLOduration=1.977643983 podStartE2EDuration="1.977643983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:49:45.966916147 +0000 UTC m=+3045.900444805" watchObservedRunningTime="2026-11-03 07:49:45.977643983 +0000 UTC m=+3045.911172641" Nov 03 07:49:46.573412 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:46.573373 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:46.585640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:46.585599 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:49:46.585942 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:46.585911 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 07:49:46.623221 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:46.623189 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:46.624500 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:46.624473 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 07:49:46.624620 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:46.624520 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 07:49:47.580483 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:47.580439 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:49:47.581146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:47.581110 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 07:49:47.622725 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:47.622691 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 07:49:47.622892 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:47.622740 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 07:49:48.580036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:48.579473 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:49:48.580036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:48.579552 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 07:49:48.622883 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:48.622846 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 07:49:48.623314 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:48.622901 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 07:49:49.573486 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:49.573442 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:49:49.573806 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:49.573773 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 07:49:49.623494 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:49.623462 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 07:49:49.623957 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:49.623929 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 07:49:50.574036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:50.573414 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:49:50.574036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:50.573479 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 07:49:50.623560 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:50.623526 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 07:49:50.623928 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:50.623580 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 07:49:51.573388 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:51.573350 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:49:51.573598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:51.573419 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 07:49:51.622945 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:51.622905 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 07:49:51.623123 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:51.622969 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 07:49:52.573293 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:52.573252 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:49:52.573679 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:52.573322 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 07:49:52.622973 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:52.622931 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 07:49:52.623175 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:52.622998 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 07:49:53.576902 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:53.576869 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:53.577318 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:53.577148 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:53.577881 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:53.577861 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" Nov 03 07:49:53.626859 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:53.626832 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:53.627098 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:53.627082 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:53.627774 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:53.627753 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" Nov 03 07:49:53.755792 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:53.755740 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podStartSLOduration=9.755726149000001 podStartE2EDuration="9.755726149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:49:46.096123488 +0000 UTC m=+3046.029652141" watchObservedRunningTime="2026-11-03 07:49:53.755726149 +0000 UTC m=+3053.689254804" Nov 03 07:49:59.380231 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.380194 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6"] Nov 03 07:49:59.383522 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.383505 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412358 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412331 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa09988a-1035-414f-8838-21f9395f72ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412365 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412403 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa09988a-1035-414f-8838-21f9395f72ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412471 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412428 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412475 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412505 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbxf\" (UniqueName: \"kubernetes.io/projected/aa09988a-1035-414f-8838-21f9395f72ee-kube-api-access-mbbxf\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412580 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412533 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa09988a-1035-414f-8838-21f9395f72ee-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412676 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412645 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.412711 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.412680 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.491327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.491292 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6"] Nov 03 07:49:59.513113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513067 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513114 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa09988a-1035-414f-8838-21f9395f72ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513146 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513176 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513208 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbxf\" (UniqueName: \"kubernetes.io/projected/aa09988a-1035-414f-8838-21f9395f72ee-kube-api-access-mbbxf\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513250 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa09988a-1035-414f-8838-21f9395f72ee-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513311 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513369 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513344 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513392 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa09988a-1035-414f-8838-21f9395f72ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513725 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513777 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513756 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513893 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513821 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.513967 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.513945 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.514152 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.514133 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa09988a-1035-414f-8838-21f9395f72ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.515898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.515860 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa09988a-1035-414f-8838-21f9395f72ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.515898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.515891 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa09988a-1035-414f-8838-21f9395f72ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.579514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.579485 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa09988a-1035-414f-8838-21f9395f72ee-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.579661 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.579594 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbxf\" (UniqueName: \"kubernetes.io/projected/aa09988a-1035-414f-8838-21f9395f72ee-kube-api-access-mbbxf\") pod \"maas-default-gateway-openshift-default-58d898bd55-rdzr6\" (UID: \"aa09988a-1035-414f-8838-21f9395f72ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:49:59.696622 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:49:59.696534 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:50:00.125636 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:00.125600 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6"] Nov 03 07:50:00.126873 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:50:00.126846 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa09988a_1035_414f_8838_21f9395f72ee.slice/crio-367e62c7332225596b0c5fe01f834f89656e1f8f00b3d736214f6671d211fd16 WatchSource:0}: Error finding container 367e62c7332225596b0c5fe01f834f89656e1f8f00b3d736214f6671d211fd16: Status 404 returned error can't find the container with id 367e62c7332225596b0c5fe01f834f89656e1f8f00b3d736214f6671d211fd16 Nov 03 07:50:00.128898 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:00.128872 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:50:00.128959 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:00.128950 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:50:00.128994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:00.128977 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 07:50:00.981458 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:00.981418 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" event={"ID":"aa09988a-1035-414f-8838-21f9395f72ee","Type":"ContainerStarted","Data":"69fe87df903f415663e1379bc9afdadf8fd251dffe61d1a5d503be73857abc78"} Nov 03 07:50:00.981868 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:00.981474 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" event={"ID":"aa09988a-1035-414f-8838-21f9395f72ee","Type":"ContainerStarted","Data":"367e62c7332225596b0c5fe01f834f89656e1f8f00b3d736214f6671d211fd16"} Nov 03 07:50:01.696996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:01.696955 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:50:01.698170 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:01.698147 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 07:50:01.698243 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:01.698210 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podUID="aa09988a-1035-414f-8838-21f9395f72ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 07:50:02.697975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:02.697935 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 07:50:02.698415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:02.697991 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podUID="aa09988a-1035-414f-8838-21f9395f72ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 07:50:03.671207 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:03.671165 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:50:03.697952 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:03.697918 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 07:50:03.698143 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:03.697981 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podUID="aa09988a-1035-414f-8838-21f9395f72ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 07:50:04.697789 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:04.697750 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 07:50:04.697976 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:04.697821 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podUID="aa09988a-1035-414f-8838-21f9395f72ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 07:50:05.618628 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:05.618580 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:50:05.697389 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:05.697349 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 07:50:05.697574 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:05.697420 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podUID="aa09988a-1035-414f-8838-21f9395f72ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 07:50:06.697914 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:06.697869 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 07:50:06.698465 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:06.697921 4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podUID="aa09988a-1035-414f-8838-21f9395f72ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 07:50:08.257409 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:08.257371 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:50:08.431625 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:08.431588 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:50:08.432140 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:08.432105 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:50:08.432974 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:08.432944 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" Nov 03 07:50:08.503822 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:08.503772 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:50:08.608034 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:08.607964 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podStartSLOduration=9.607952092 podStartE2EDuration="9.607952092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:50:01.170371922 +0000 UTC m=+3061.103900582" watchObservedRunningTime="2026-11-03 07:50:08.607952092 +0000 UTC m=+3068.541480786" Nov 03 07:50:09.139177 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:50:09.139140 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:50:15.267617 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:15.267577 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:50:15.503597 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:15.503563 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:50:39.138755 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:50:39.138715 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:50:49.006309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:50:49.006212 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:51:08.595041 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:51:08.594979 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:51:09.137725 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:51:09.137679 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:51:10.470686 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:51:10.470653 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:51:21.595626 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:51:21.595589 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:51:39.137706 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:51:39.137669 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:51:41.975113 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:51:41.975075 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:51:59.082573 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:51:59.082533 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:52:08.964117 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:52:08.964079 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:52:09.138067 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:52:09.138033 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:52:30.177960 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:52:30.177881 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:52:39.137953 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:52:39.137916 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:52:40.902324 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:52:40.902288 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:52:53.582115 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:52:53.582076 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:53:09.138294 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:53:09.138259 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:53:09.294449 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:53:09.294408 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:53:09.490259 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:53:09.490164 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:53:15.812328 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:53:15.812296 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:53:20.744553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:53:20.744502 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:53:25.588434 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:53:25.588396 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:53:39.137319 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:53:39.137281 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:53:43.907415 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:53:43.907339 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:53:56.710748 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:53:56.710707 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:54:00.063296 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:00.063261 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72"] Nov 03 07:54:00.115026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:00.114972 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e423dd-fjb72"] Nov 03 07:54:00.190187 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:00.190154 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427024d8-76f8-4759-8018-3deb63b09fbc" path="/var/lib/kubelet/pods/427024d8-76f8-4759-8018-3deb63b09fbc/volumes" Nov 03 07:54:01.926766 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:01.926738 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:54:01.930446 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:01.930421 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:54:01.936856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:01.936834 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:54:01.940300 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:01.940282 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:54:09.136955 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:54:09.136916 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:54:10.986164 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:10.986130 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:54:16.492896 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:16.492858 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:54:22.588579 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:22.588534 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:54:38.418947 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:38.418912 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:54:39.138719 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:54:39.138684 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:54:39.739940 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:39.739900 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:54:51.766292 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:51.766255 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:54:54.476156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:54.476116 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:54:54.476818 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:54.476772 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6" podUID="aa09988a-1035-414f-8838-21f9395f72ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:54:54.620172 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:54.620132 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:54:54.620381 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:54.620216 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:54:54.669174 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:54.669132 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:54:54.669365 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:54:54.669195 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:55:02.345771 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:55:02.345737 4925 scope.go:122] "RemoveContainer" containerID="baab99d50e7723ae08b67ab6c07d1703b380245da26028e7a177d0de805ad350" Nov 03 07:55:02.354924 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:55:02.354905 4925 scope.go:122] "RemoveContainer" containerID="7f050cee72155ed3e3615cb922b0ce60fcc86ac3e68ea7bf829df0fadea588db" Nov 03 07:55:02.570145 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:55:02.570108 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:55:09.137357 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:55:09.137321 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:55:09.986278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:55:09.986237 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:55:25.387814 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:55:25.387774 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:55:35.690804 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:55:35.690769 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:55:39.138921 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:55:39.138879 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:55:54.067289 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:55:54.067245 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:56:09.136741 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:56:09.136693 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:56:10.392390 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:56:10.392354 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:56:17.192278 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:56:17.192242 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:56:18.039397 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:56:18.039357 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:56:27.905771 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:56:27.905735 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:56:39.138305 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:56:39.138266 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:56:48.697640 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:56:48.697540 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:56:59.397488 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:56:59.397449 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:57:05.919551 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:05.919507 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:06.544985 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:06.544948 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:06.789629 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:06.789588 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:09.137225 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:57:09.137187 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:57:10.441535 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:10.441502 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:10.689955 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:10.689917 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:13.743878 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:13.743838 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:13.983935 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:13.983901 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:17.014954 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:17.014919 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:17.215090 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:17.215055 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:22.263798 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:22.263766 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:25.188239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:25.188204 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:57:27.491894 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:27.491861 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:32.668455 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:32.668416 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:35.272834 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:35.272802 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:35.871215 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:35.871181 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:36.128653 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:36.128543 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:38.090327 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:38.090274 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:39.136714 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:57:39.136678 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:57:39.444139 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.444048 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng"] Nov 03 07:57:39.447667 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.447641 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.507037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.506981 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng"] Nov 03 07:57:39.555440 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.555405 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-98d4f3\"/\"kube-root-ca.crt\"" Nov 03 07:57:39.555440 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.555405 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-98d4f3\"/\"openshift-service-ca.crt\"" Nov 03 07:57:39.555633 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.555589 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-98d4f3\"/\"simulator-single-main-98d4f3-kserve-self-signed-certs\"" Nov 03 07:57:39.555670 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.555641 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-98d4f3\"/\"default-dockercfg-tk9dl\"" Nov 03 07:57:39.566387 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.566362 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-model-cache\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.566541 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.566395 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzlzw\" (UniqueName: \"kubernetes.io/projected/2b6bc653-f354-44ee-a869-86e0c54b6ac1-kube-api-access-rzlzw\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.566541 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.566414 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-dshm\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.566541 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.566435 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tmp-dir\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.566681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.566634 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tls-certs\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.566681 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.566652 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-home\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.586840 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.586815 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:39.667737 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.667705 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tmp-dir\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.667914 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.667873 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tls-certs\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.667914 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.667906 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-home\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.668036 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.667997 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-model-cache\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.668092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.668057 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlzw\" (UniqueName: \"kubernetes.io/projected/2b6bc653-f354-44ee-a869-86e0c54b6ac1-kube-api-access-rzlzw\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.668092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.668082 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-dshm\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.668194 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.668157 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tmp-dir\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.668372 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.668348 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-home\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.668454 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.668393 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-model-cache\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.670491 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.670461 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-dshm\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.670612 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.670593 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tls-certs\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.732156 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.732095 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlzw\" (UniqueName: \"kubernetes.io/projected/2b6bc653-f354-44ee-a869-86e0c54b6ac1-kube-api-access-rzlzw\") pod \"simulator-single-main-98d4f3-kserve-74d7947544-bd2ng\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:39.758911 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:39.758884 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:40.141026 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:40.140975 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng"] Nov 03 07:57:40.145875 ip-10-0-136-184 kubenswrapper[4925]: W1103 07:57:40.145833 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6bc653_f354_44ee_a869_86e0c54b6ac1.slice/crio-79dbd536dcfbe22fe7d3dc46da9ac328303ef0fbab4dc4c03ad4da267f6d7e9b WatchSource:0}: Error finding container 79dbd536dcfbe22fe7d3dc46da9ac328303ef0fbab4dc4c03ad4da267f6d7e9b: Status 404 returned error can't find the container with id 79dbd536dcfbe22fe7d3dc46da9ac328303ef0fbab4dc4c03ad4da267f6d7e9b Nov 03 07:57:40.572162 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:40.572127 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" event={"ID":"2b6bc653-f354-44ee-a869-86e0c54b6ac1","Type":"ContainerStarted","Data":"d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735"} Nov 03 07:57:40.572333 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:40.572168 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" event={"ID":"2b6bc653-f354-44ee-a869-86e0c54b6ac1","Type":"ContainerStarted","Data":"79dbd536dcfbe22fe7d3dc46da9ac328303ef0fbab4dc4c03ad4da267f6d7e9b"} Nov 03 07:57:40.693504 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:40.693455 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" podStartSLOduration=1.693441631 podStartE2EDuration="1.693441631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:40.692210716 +0000 UTC m=+3520.625739372" watchObservedRunningTime="2026-11-03 07:57:40.693441631 +0000 UTC m=+3520.626970287" Nov 03 07:57:40.759528 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:40.759493 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:43.264780 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:43.264746 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:44.197669 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:44.197631 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:57:48.544316 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:48.544267 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:49.759440 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:49.759406 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:49.771862 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:49.771835 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:50.634619 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:50.634585 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:57:53.691576 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:53.691533 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:55.453295 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:55.453263 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:57:58.760376 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:57:58.760335 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:03.860553 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:03.860518 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:09.138321 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:58:09.138281 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:58:09.216476 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:09.216442 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:09.389462 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:09.389382 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:58:11.033623 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:11.033541 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:14.063421 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:14.063380 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:19.161660 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:19.161621 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:24.266056 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:24.265988 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:29.468944 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:29.468897 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:34.654314 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:34.654251 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:35.184856 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:35.184820 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:38.375473 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:38.375437 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:58:39.138640 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:58:39.138604 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:58:40.095115 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:40.095079 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng"] Nov 03 07:58:40.224298 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:40.224266 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:40.467234 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:40.467132 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:45.469962 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:45.469919 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:47.618220 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:47.618182 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:47.816931 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:47.816892 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:50.003968 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:50.003927 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:51.628849 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:51.628814 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:51.919309 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:51.919221 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:52.697239 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:52.697204 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:54.673450 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:54.673410 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:58:55.115414 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.115370 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" podUID="2b6bc653-f354-44ee-a869-86e0c54b6ac1" containerName="main" containerID="cri-o://d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735" gracePeriod=45 Nov 03 07:58:55.363889 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.363864 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:58:55.447501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447419 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tls-certs\") pod \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " Nov 03 07:58:55.447501 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447472 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6bc653-f354-44ee-a869-86e0c54b6ac1-kube-api-access-rzlzw\" (UniqueName: \"kubernetes.io/projected/2b6bc653-f354-44ee-a869-86e0c54b6ac1-kube-api-access-rzlzw\") pod \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " Nov 03 07:58:55.447726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447608 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-home\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-home\") pod \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " Nov 03 07:58:55.447726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447643 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-model-cache\") pod \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " Nov 03 07:58:55.447726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447669 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tmp-dir\") pod \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " Nov 03 07:58:55.447726 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447699 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-dshm\") pod \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\" (UID: \"2b6bc653-f354-44ee-a869-86e0c54b6ac1\") " Nov 03 07:58:55.447923 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447882 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-home" pod "2b6bc653-f354-44ee-a869-86e0c54b6ac1" (UID: "2b6bc653-f354-44ee-a869-86e0c54b6ac1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:58:55.447975 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447901 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-model-cache" pod "2b6bc653-f354-44ee-a869-86e0c54b6ac1" (UID: "2b6bc653-f354-44ee-a869-86e0c54b6ac1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:58:55.448065 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.447977 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tmp-dir" pod "2b6bc653-f354-44ee-a869-86e0c54b6ac1" (UID: "2b6bc653-f354-44ee-a869-86e0c54b6ac1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:58:55.448129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.448080 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:55.448129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.448093 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:55.448129 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.448106 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:55.449858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.449832 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tls-certs" pod "2b6bc653-f354-44ee-a869-86e0c54b6ac1" (UID: "2b6bc653-f354-44ee-a869-86e0c54b6ac1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:58:55.449858 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.449838 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-dshm" pod "2b6bc653-f354-44ee-a869-86e0c54b6ac1" (UID: "2b6bc653-f354-44ee-a869-86e0c54b6ac1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:58:55.449996 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.449885 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6bc653-f354-44ee-a869-86e0c54b6ac1-kube-api-access-rzlzw" pod "2b6bc653-f354-44ee-a869-86e0c54b6ac1" (UID: "2b6bc653-f354-44ee-a869-86e0c54b6ac1"). InnerVolumeSpecName "kube-api-access-rzlzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:58:55.549238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.549207 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6bc653-f354-44ee-a869-86e0c54b6ac1-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:55.549238 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.549237 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzlzw\" (UniqueName: \"kubernetes.io/projected/2b6bc653-f354-44ee-a869-86e0c54b6ac1-kube-api-access-rzlzw\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:55.549424 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.549249 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6bc653-f354-44ee-a869-86e0c54b6ac1-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:55.958506 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.958468 4925 generic.go:356] "Generic (PLEG): container finished" podID="2b6bc653-f354-44ee-a869-86e0c54b6ac1" containerID="d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735" exitCode=0 Nov 03 07:58:55.958994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.958522 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" Nov 03 07:58:55.958994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.958567 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" event={"ID":"2b6bc653-f354-44ee-a869-86e0c54b6ac1","Type":"ContainerDied","Data":"d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735"} Nov 03 07:58:55.958994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.958621 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng" event={"ID":"2b6bc653-f354-44ee-a869-86e0c54b6ac1","Type":"ContainerDied","Data":"79dbd536dcfbe22fe7d3dc46da9ac328303ef0fbab4dc4c03ad4da267f6d7e9b"} Nov 03 07:58:55.958994 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.958643 4925 scope.go:122] "RemoveContainer" containerID="d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735" Nov 03 07:58:55.969030 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.968999 4925 scope.go:122] "RemoveContainer" containerID="d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735" Nov 03 07:58:55.969378 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:58:55.969354 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735\": container with ID starting with d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735 not found: ID does not exist" containerID="d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735" Nov 03 07:58:55.969439 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:55.969387 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735"} err="failed to get container status \"d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735\": rpc error: code = NotFound desc = could not find container \"d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735\": container with ID starting with d0d5d825076bbd10cb710c869f3f94dcac170aff4a6c31ea4d113afeb3c32735 not found: ID does not exist" Nov 03 07:58:56.129092 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:56.129053 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng"] Nov 03 07:58:56.179719 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:56.179684 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-98d4f3/simulator-single-main-98d4f3-kserve-74d7947544-bd2ng"] Nov 03 07:58:56.190349 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:56.190314 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6bc653-f354-44ee-a869-86e0c54b6ac1" path="/var/lib/kubelet/pods/2b6bc653-f354-44ee-a869-86e0c54b6ac1/volumes" Nov 03 07:58:59.362401 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:59.362362 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:59.553603 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:59.553563 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:58:59.740710 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:58:59.740618 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:59:01.962467 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:01.962438 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:59:01.966207 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:01.966186 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:59:01.973723 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:01.973704 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 07:59:01.977276 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:01.977259 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 07:59:09.137615 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:59:09.137579 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 07:59:11.376223 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:11.376177 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:59:16.098101 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:16.098067 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:59:17.796037 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:17.773135 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:59:18.606270 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:18.606233 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:59:24.974216 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:24.974175 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 07:59:26.799727 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:26.799690 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 07:59:35.998514 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:35.998479 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh"] Nov 03 07:59:35.999087 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:35.998673 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" podUID="4b816edd-5ac9-44ec-86f9-f50b6e5dc545" containerName="manager" containerID="cri-o://36f69426dcca3cb3407f000efcb2dd88392cd06b9a083bf455bdb75701383db8" gracePeriod=10 Nov 03 07:59:36.204043 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.203991 4925 generic.go:356] "Generic (PLEG): container finished" podID="4b816edd-5ac9-44ec-86f9-f50b6e5dc545" containerID="36f69426dcca3cb3407f000efcb2dd88392cd06b9a083bf455bdb75701383db8" exitCode=0 Nov 03 07:59:36.204213 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.204039 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" event={"ID":"4b816edd-5ac9-44ec-86f9-f50b6e5dc545","Type":"ContainerDied","Data":"36f69426dcca3cb3407f000efcb2dd88392cd06b9a083bf455bdb75701383db8"} Nov 03 07:59:36.255891 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.255840 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:59:36.394720 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.394691 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-kube-api-access-rhhgh\" (UniqueName: \"kubernetes.io/projected/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-kube-api-access-rhhgh\") pod \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " Nov 03 07:59:36.394876 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.394786 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-extensions-socket-volume\") pod \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\" (UID: \"4b816edd-5ac9-44ec-86f9-f50b6e5dc545\") " Nov 03 07:59:36.395274 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.395243 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-extensions-socket-volume" pod "4b816edd-5ac9-44ec-86f9-f50b6e5dc545" (UID: "4b816edd-5ac9-44ec-86f9-f50b6e5dc545"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:59:36.397163 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.397130 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-kube-api-access-rhhgh" pod "4b816edd-5ac9-44ec-86f9-f50b6e5dc545" (UID: "4b816edd-5ac9-44ec-86f9-f50b6e5dc545"). InnerVolumeSpecName "kube-api-access-rhhgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:59:36.496146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.496109 4925 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-extensions-socket-volume\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:36.496146 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:36.496143 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhgh\" (UniqueName: \"kubernetes.io/projected/4b816edd-5ac9-44ec-86f9-f50b6e5dc545-kube-api-access-rhhgh\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:37.210302 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:37.210267 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" Nov 03 07:59:37.210747 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:37.210319 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh" event={"ID":"4b816edd-5ac9-44ec-86f9-f50b6e5dc545","Type":"ContainerDied","Data":"1512484978a7e5be2fc9f29898bd4c31065e4a655661149f374fab2a8a956b96"} Nov 03 07:59:37.210747 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:37.210379 4925 scope.go:122] "RemoveContainer" containerID="36f69426dcca3cb3407f000efcb2dd88392cd06b9a083bf455bdb75701383db8" Nov 03 07:59:37.435125 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:37.435088 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh"] Nov 03 07:59:37.486598 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:37.486523 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6bzgh"] Nov 03 07:59:38.196203 ip-10-0-136-184 kubenswrapper[4925]: I1103 07:59:38.196161 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b816edd-5ac9-44ec-86f9-f50b6e5dc545" path="/var/lib/kubelet/pods/4b816edd-5ac9-44ec-86f9-f50b6e5dc545/volumes" Nov 03 07:59:39.137195 ip-10-0-136-184 kubenswrapper[4925]: E1103 07:59:39.137147 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:00:09.136629 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:00:09.136586 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:00:11.778348 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:11.778295 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:00:14.886738 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:14.886700 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:00:15.127289 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:15.127246 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:00:25.199547 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:25.199506 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:00:25.399481 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:25.399448 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:00:39.137116 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:00:39.137081 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:00:39.600427 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.600395 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4"] Nov 03 08:00:39.600781 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.600767 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2b6bc653-f354-44ee-a869-86e0c54b6ac1" containerName="main" Nov 03 08:00:39.600837 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.600782 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6bc653-f354-44ee-a869-86e0c54b6ac1" containerName="main" Nov 03 08:00:39.600837 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.600807 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4b816edd-5ac9-44ec-86f9-f50b6e5dc545" containerName="manager" Nov 03 08:00:39.600837 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.600812 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b816edd-5ac9-44ec-86f9-f50b6e5dc545" containerName="manager" Nov 03 08:00:39.600950 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.600876 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6bc653-f354-44ee-a869-86e0c54b6ac1" containerName="main" Nov 03 08:00:39.600950 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.600887 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b816edd-5ac9-44ec-86f9-f50b6e5dc545" containerName="manager" Nov 03 08:00:39.603554 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.603535 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:39.708916 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.708883 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ckjh4\"" Nov 03 08:00:39.711132 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.711106 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4"] Nov 03 08:00:39.743105 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.743080 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404a8fbd-a76d-4d18-b82e-23e1ace2d745-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fm5w4\" (UID: \"404a8fbd-a76d-4d18-b82e-23e1ace2d745\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:39.743212 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.743118 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zd5\" (UniqueName: \"kubernetes.io/projected/404a8fbd-a76d-4d18-b82e-23e1ace2d745-kube-api-access-g9zd5\") pod \"kuadrant-operator-controller-manager-678d87766f-fm5w4\" (UID: \"404a8fbd-a76d-4d18-b82e-23e1ace2d745\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:39.844068 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.844043 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404a8fbd-a76d-4d18-b82e-23e1ace2d745-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fm5w4\" (UID: \"404a8fbd-a76d-4d18-b82e-23e1ace2d745\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:39.844173 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.844075 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zd5\" (UniqueName: \"kubernetes.io/projected/404a8fbd-a76d-4d18-b82e-23e1ace2d745-kube-api-access-g9zd5\") pod \"kuadrant-operator-controller-manager-678d87766f-fm5w4\" (UID: \"404a8fbd-a76d-4d18-b82e-23e1ace2d745\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:39.844420 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.844402 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404a8fbd-a76d-4d18-b82e-23e1ace2d745-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fm5w4\" (UID: \"404a8fbd-a76d-4d18-b82e-23e1ace2d745\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:39.908485 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.908419 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zd5\" (UniqueName: \"kubernetes.io/projected/404a8fbd-a76d-4d18-b82e-23e1ace2d745-kube-api-access-g9zd5\") pod \"kuadrant-operator-controller-manager-678d87766f-fm5w4\" (UID: \"404a8fbd-a76d-4d18-b82e-23e1ace2d745\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:39.915831 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:39.915805 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:40.348989 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:40.348958 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4"] Nov 03 08:00:40.350873 ip-10-0-136-184 kubenswrapper[4925]: W1103 08:00:40.350847 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404a8fbd_a76d_4d18_b82e_23e1ace2d745.slice/crio-61ea17367bcca376f277c89ed1f7525d6c18f676f1702fe55853b656d51587cd WatchSource:0}: Error finding container 61ea17367bcca376f277c89ed1f7525d6c18f676f1702fe55853b656d51587cd: Status 404 returned error can't find the container with id 61ea17367bcca376f277c89ed1f7525d6c18f676f1702fe55853b656d51587cd Nov 03 08:00:40.521135 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:40.521101 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" event={"ID":"404a8fbd-a76d-4d18-b82e-23e1ace2d745","Type":"ContainerStarted","Data":"e0a9e6837831c34e13e4aa299f2e759f979ad675c5e6ad3530b65cc9c8946c0c"} Nov 03 08:00:40.521302 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:40.521152 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" event={"ID":"404a8fbd-a76d-4d18-b82e-23e1ace2d745","Type":"ContainerStarted","Data":"61ea17367bcca376f277c89ed1f7525d6c18f676f1702fe55853b656d51587cd"} Nov 03 08:00:40.521302 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:40.521196 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:00:40.644582 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:40.644492 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" podStartSLOduration=1.644478798 podStartE2EDuration="1.644478798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:40.643706008 +0000 UTC m=+3700.577234658" watchObservedRunningTime="2026-11-03 08:00:40.644478798 +0000 UTC m=+3700.578007453" Nov 03 08:00:51.528059 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:00:51.527993 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fm5w4" Nov 03 08:01:00.142586 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.142532 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:00.150653 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.150623 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv"] Nov 03 08:01:00.154334 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.154310 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.212093 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.212061 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv"] Nov 03 08:01:00.217248 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.217218 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/77206c00-a1b5-48e9-a136-c87966d96a42-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.217390 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.217265 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgnw\" (UniqueName: \"kubernetes.io/projected/77206c00-a1b5-48e9-a136-c87966d96a42-kube-api-access-tfgnw\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.217443 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.217388 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/77206c00-a1b5-48e9-a136-c87966d96a42-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.253699 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.253672 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 08:01:00.260683 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.260662 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 08:01:00.318422 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.318395 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/77206c00-a1b5-48e9-a136-c87966d96a42-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.318561 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.318435 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgnw\" (UniqueName: \"kubernetes.io/projected/77206c00-a1b5-48e9-a136-c87966d96a42-kube-api-access-tfgnw\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.318561 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.318480 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/77206c00-a1b5-48e9-a136-c87966d96a42-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.319144 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.319124 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/77206c00-a1b5-48e9-a136-c87966d96a42-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.320926 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.320905 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/77206c00-a1b5-48e9-a136-c87966d96a42-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.331683 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.331643 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:00.372005 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.371979 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:01:00.382453 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.382430 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgnw\" (UniqueName: \"kubernetes.io/projected/77206c00-a1b5-48e9-a136-c87966d96a42-kube-api-access-tfgnw\") pod \"kuadrant-console-plugin-66c4fb575d-chsvv\" (UID: \"77206c00-a1b5-48e9-a136-c87966d96a42\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.473102 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.472997 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" Nov 03 08:01:00.775354 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.775258 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:00.868061 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.867831 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv"] Nov 03 08:01:00.868269 ip-10-0-136-184 kubenswrapper[4925]: W1103 08:01:00.868188 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77206c00_a1b5_48e9_a136_c87966d96a42.slice/crio-75e5c3138ada1e34d0d0ff9a91fffc5acc718319ff505c5cbd6781b1a90f550e WatchSource:0}: Error finding container 75e5c3138ada1e34d0d0ff9a91fffc5acc718319ff505c5cbd6781b1a90f550e: Status 404 returned error can't find the container with id 75e5c3138ada1e34d0d0ff9a91fffc5acc718319ff505c5cbd6781b1a90f550e Nov 03 08:01:00.869637 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:00.869613 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:01:01.633078 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:01.633038 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" event={"ID":"77206c00-a1b5-48e9-a136-c87966d96a42","Type":"ContainerStarted","Data":"75e5c3138ada1e34d0d0ff9a91fffc5acc718319ff505c5cbd6781b1a90f550e"} Nov 03 08:01:07.677447 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:07.677411 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" event={"ID":"77206c00-a1b5-48e9-a136-c87966d96a42","Type":"ContainerStarted","Data":"f72a3ea25009c33a7b6a0ab1639af32ab625c966ac85b9a75ac5bfe35375f043"} Nov 03 08:01:07.799701 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:07.799646 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-chsvv" podStartSLOduration=1.528268603 podStartE2EDuration="7.799631467s" totalImagesPullingTime="6.271362864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:07.797534759 +0000 UTC m=+3727.731063416" watchObservedRunningTime="2026-11-03 08:01:07.799631467 +0000 UTC m=+3727.733160122" Nov 03 08:01:09.138564 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:01:09.138529 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:01:12.072938 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:12.072858 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:14.860672 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:14.860638 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:15.047336 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:15.047294 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:17.593819 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:17.593773 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:01:24.624692 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:24.624644 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:01:24.625243 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:24.624738 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:01:24.674190 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:24.674147 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:01:24.674510 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:24.674460 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:01:25.668413 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:25.668377 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:01:39.138373 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:01:39.138331 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:01:46.186628 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:46.186587 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:46.581876 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:46.581840 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:01:46.910271 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:46.910162 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:47.280362 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:47.280274 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:50.886884 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:50.886847 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:51.151715 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:51.151622 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:54.190509 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:54.190477 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:54.429768 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:54.429729 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:57.392026 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:57.391958 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:57.599872 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:57.599832 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:01:57.801958 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:01:57.801866 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:02.750610 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:02.750570 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:05.591324 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:05.591290 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:02:08.002219 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:08.002181 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:09.137844 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:02:09.137809 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:02:12.421152 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:12.421118 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:13.164394 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:13.164347 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:15.676297 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:15.676254 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:16.277509 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:16.277466 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:16.464209 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:16.464173 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:18.779907 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:18.779869 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:19.219039 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.218987 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz"] Nov 03 08:02:19.222713 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.222690 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.227050 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.227000 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tmp-dir\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.227175 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.227157 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-dshm\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.227237 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.227209 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-model-cache\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.227444 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.227278 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tls-certs\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.227444 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.227403 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cch48\" (UniqueName: \"kubernetes.io/projected/d9ad6cbd-bd0a-4666-9bec-e31f44615783-kube-api-access-cch48\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.227444 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.227433 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-home\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.328496 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328454 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b435d8\"/\"default-dockercfg-flzwh\"" Nov 03 08:02:19.328496 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328460 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b435d8\"/\"kube-root-ca.crt\"" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328498 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b435d8\"/\"openshift-service-ca.crt\"" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328558 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-dshm\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328579 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b435d8\"/\"simulator-single-main-b435d8-kserve-self-signed-certs\"" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328601 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-model-cache\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328622 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tls-certs\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328654 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cch48\" (UniqueName: \"kubernetes.io/projected/d9ad6cbd-bd0a-4666-9bec-e31f44615783-kube-api-access-cch48\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328685 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-home\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.328808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.328718 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tmp-dir\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.329282 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.329096 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-home\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.329282 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.329113 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-model-cache\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.329282 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.329172 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tmp-dir\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.331176 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.331147 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-dshm\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.331348 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.331331 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tls-certs\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.334962 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.332768 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz"] Nov 03 08:02:19.360182 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.360123 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:19.392740 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.392707 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cch48\" (UniqueName: \"kubernetes.io/projected/d9ad6cbd-bd0a-4666-9bec-e31f44615783-kube-api-access-cch48\") pod \"simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.534369 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.534275 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:19.919566 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:19.919534 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz"] Nov 03 08:02:19.920772 ip-10-0-136-184 kubenswrapper[4925]: W1103 08:02:19.920746 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ad6cbd_bd0a_4666_9bec_e31f44615783.slice/crio-54006d93ca5f0514fa1f4e756066fd08405b6dc9ce828e99a7d4bff3c40f2001 WatchSource:0}: Error finding container 54006d93ca5f0514fa1f4e756066fd08405b6dc9ce828e99a7d4bff3c40f2001: Status 404 returned error can't find the container with id 54006d93ca5f0514fa1f4e756066fd08405b6dc9ce828e99a7d4bff3c40f2001 Nov 03 08:02:20.087578 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:20.087543 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" event={"ID":"d9ad6cbd-bd0a-4666-9bec-e31f44615783","Type":"ContainerStarted","Data":"4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0"} Nov 03 08:02:20.087781 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:20.087589 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" event={"ID":"d9ad6cbd-bd0a-4666-9bec-e31f44615783","Type":"ContainerStarted","Data":"54006d93ca5f0514fa1f4e756066fd08405b6dc9ce828e99a7d4bff3c40f2001"} Nov 03 08:02:20.210253 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:20.210142 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" podStartSLOduration=1.210128122 podStartE2EDuration="1.210128122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:20.208319928 +0000 UTC m=+3800.141848584" watchObservedRunningTime="2026-11-03 08:02:20.210128122 +0000 UTC m=+3800.143656779" Nov 03 08:02:20.535328 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:20.535248 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:23.870483 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:23.870448 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:02:23.968444 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:23.968391 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:29.074625 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:29.074584 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:29.535405 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:29.535321 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:29.547732 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:29.547704 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:30.150770 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:30.150740 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:02:34.156523 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:34.156476 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:39.137893 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:02:39.137858 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:02:39.293306 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:39.293252 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:44.361769 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:44.361673 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:49.467304 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:49.467268 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:49.667554 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:49.667518 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:02:54.558737 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:54.558696 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:02:59.757082 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:02:59.757046 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:04.868190 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:04.868149 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:09.137683 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:03:09.137646 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:03:10.061799 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:10.061763 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:12.755099 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:12.755046 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:15.198301 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:15.198267 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:15.898692 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:15.898648 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:17.772658 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:17.772623 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:03:20.676366 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:20.676327 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:20.803566 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:20.803529 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz"] Nov 03 08:03:21.017109 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:21.017002 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:25.958382 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:25.958337 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:28.320563 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:28.320524 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:28.512467 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:28.512432 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:30.467701 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:30.467658 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:35.157963 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:35.157931 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:03:35.824103 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:35.824060 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" podUID="d9ad6cbd-bd0a-4666-9bec-e31f44615783" containerName="main" containerID="cri-o://4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0" gracePeriod=45 Nov 03 08:03:36.075099 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.075033 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:03:36.242297 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242269 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ad6cbd-bd0a-4666-9bec-e31f44615783-kube-api-access-cch48\" (UniqueName: \"kubernetes.io/projected/d9ad6cbd-bd0a-4666-9bec-e31f44615783-kube-api-access-cch48\") pod \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " Nov 03 08:03:36.242758 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242350 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-home\") pod \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " Nov 03 08:03:36.242758 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242405 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-dshm\") pod \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " Nov 03 08:03:36.242758 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242467 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tmp-dir\") pod \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " Nov 03 08:03:36.242758 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242513 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-model-cache\") pod \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " Nov 03 08:03:36.242758 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242544 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tls-certs\") pod \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\" (UID: \"d9ad6cbd-bd0a-4666-9bec-e31f44615783\") " Nov 03 08:03:36.242758 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242675 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-home" pod "d9ad6cbd-bd0a-4666-9bec-e31f44615783" (UID: "d9ad6cbd-bd0a-4666-9bec-e31f44615783"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:36.243046 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.242787 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tmp-dir" pod "d9ad6cbd-bd0a-4666-9bec-e31f44615783" (UID: "d9ad6cbd-bd0a-4666-9bec-e31f44615783"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:36.243119 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.243085 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-model-cache" pod "d9ad6cbd-bd0a-4666-9bec-e31f44615783" (UID: "d9ad6cbd-bd0a-4666-9bec-e31f44615783"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:36.243200 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.243179 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:36.243291 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.243206 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:36.244657 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.244627 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ad6cbd-bd0a-4666-9bec-e31f44615783-kube-api-access-cch48" pod "d9ad6cbd-bd0a-4666-9bec-e31f44615783" (UID: "d9ad6cbd-bd0a-4666-9bec-e31f44615783"). InnerVolumeSpecName "kube-api-access-cch48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:03:36.244817 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.244800 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-dshm" pod "d9ad6cbd-bd0a-4666-9bec-e31f44615783" (UID: "d9ad6cbd-bd0a-4666-9bec-e31f44615783"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:36.244894 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.244864 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tls-certs" pod "d9ad6cbd-bd0a-4666-9bec-e31f44615783" (UID: "d9ad6cbd-bd0a-4666-9bec-e31f44615783"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:03:36.344483 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.344397 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:36.344483 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.344435 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad6cbd-bd0a-4666-9bec-e31f44615783-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:36.344483 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.344450 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cch48\" (UniqueName: \"kubernetes.io/projected/d9ad6cbd-bd0a-4666-9bec-e31f44615783-kube-api-access-cch48\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:36.344483 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.344464 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad6cbd-bd0a-4666-9bec-e31f44615783-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:36.477111 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.477074 4925 generic.go:356] "Generic (PLEG): container finished" podID="d9ad6cbd-bd0a-4666-9bec-e31f44615783" containerID="4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0" exitCode=0 Nov 03 08:03:36.477296 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.477123 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" Nov 03 08:03:36.477296 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.477148 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" event={"ID":"d9ad6cbd-bd0a-4666-9bec-e31f44615783","Type":"ContainerDied","Data":"4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0"} Nov 03 08:03:36.477296 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.477196 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz" event={"ID":"d9ad6cbd-bd0a-4666-9bec-e31f44615783","Type":"ContainerDied","Data":"54006d93ca5f0514fa1f4e756066fd08405b6dc9ce828e99a7d4bff3c40f2001"} Nov 03 08:03:36.477296 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.477220 4925 scope.go:122] "RemoveContainer" containerID="4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0" Nov 03 08:03:36.486587 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.486568 4925 scope.go:122] "RemoveContainer" containerID="4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0" Nov 03 08:03:36.486880 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:03:36.486856 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0\": container with ID starting with 4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0 not found: ID does not exist" containerID="4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0" Nov 03 08:03:36.486931 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.486889 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0"} err="failed to get container status \"4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0\": rpc error: code = NotFound desc = could not find container \"4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0\": container with ID starting with 4d80e20bda11d7002300725e1df78509dbf32b517c33b520fe4c08f32c8f70d0 not found: ID does not exist" Nov 03 08:03:36.647800 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.647714 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz"] Nov 03 08:03:36.698575 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:36.698539 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b435d8/simulator-single-main-b435d8-kserve-66cbdbc564-t2xfz"] Nov 03 08:03:38.123072 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:38.123030 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:38.189358 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:38.189326 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ad6cbd-bd0a-4666-9bec-e31f44615783" path="/var/lib/kubelet/pods/d9ad6cbd-bd0a-4666-9bec-e31f44615783/volumes" Nov 03 08:03:38.408997 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:38.408901 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:39.138036 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:03:39.137972 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:03:39.184302 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:39.184271 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:45.775941 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:45.775895 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:46.027961 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:46.027867 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:46.219219 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:46.219182 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:53.554041 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:53.553869 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:57.209687 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:57.209575 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:03:58.098461 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:58.098428 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:03:59.895729 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:03:59.895687 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:04:01.999475 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:01.999440 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 08:04:02.003338 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:02.003315 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 08:04:02.011224 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:02.011201 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 08:04:02.014658 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:02.014639 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 08:04:09.137109 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:04:09.137073 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:04:13.054944 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:13.054856 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:04:13.248579 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:13.248536 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:04:25.485207 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:25.485167 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:04:38.682321 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.682277 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss"] Nov 03 08:04:38.682841 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.682821 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d9ad6cbd-bd0a-4666-9bec-e31f44615783" containerName="main" Nov 03 08:04:38.682926 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.682845 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ad6cbd-bd0a-4666-9bec-e31f44615783" containerName="main" Nov 03 08:04:38.683005 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.682991 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ad6cbd-bd0a-4666-9bec-e31f44615783" containerName="main" Nov 03 08:04:38.685966 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.685946 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.742095 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.742062 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0730fa14-kserve-self-signed-certs\"" Nov 03 08:04:38.742251 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.742126 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vdz8z\"" Nov 03 08:04:38.791703 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.791670 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 08:04:38.791861 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.791746 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 08:04:38.794038 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.794001 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss"] Nov 03 08:04:38.830460 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.830423 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:04:38.843085 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.843055 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tmp-dir\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.843257 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.843106 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tls-certs\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.843257 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.843211 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-model-cache\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.843257 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.843245 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4ntn\" (UniqueName: \"kubernetes.io/projected/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-kube-api-access-h4ntn\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.843393 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.843324 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-home\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.843393 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.843356 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-dshm\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.944502 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.944422 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-dshm\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.944502 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.944468 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tmp-dir\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.944502 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.944499 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tls-certs\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.944799 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.944534 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-model-cache\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.944799 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.944555 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4ntn\" (UniqueName: \"kubernetes.io/projected/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-kube-api-access-h4ntn\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.944799 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.944603 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-home\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.944954 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.944933 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tmp-dir\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.945043 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.945003 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-model-cache\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.945107 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.945042 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-home\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.946841 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.946815 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-dshm\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:38.947256 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:38.947237 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tls-certs\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:39.009059 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:39.009005 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4ntn\" (UniqueName: \"kubernetes.io/projected/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-kube-api-access-h4ntn\") pod \"e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:39.138669 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:04:39.138627 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:04:39.297172 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:39.297088 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:39.678089 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:39.678059 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss"] Nov 03 08:04:39.678909 ip-10-0-136-184 kubenswrapper[4925]: W1103 08:04:39.678882 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod605349c2_1d64_43f0_afb2_9b2ad6e5a7c9.slice/crio-9eebfa9157f101126d4dad6d294bbb429f1e66bc25ff97df5f6b20f960e21e68 WatchSource:0}: Error finding container 9eebfa9157f101126d4dad6d294bbb429f1e66bc25ff97df5f6b20f960e21e68: Status 404 returned error can't find the container with id 9eebfa9157f101126d4dad6d294bbb429f1e66bc25ff97df5f6b20f960e21e68 Nov 03 08:04:39.845808 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:39.845768 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" event={"ID":"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9","Type":"ContainerStarted","Data":"ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8"} Nov 03 08:04:39.846267 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:39.845820 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" event={"ID":"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9","Type":"ContainerStarted","Data":"9eebfa9157f101126d4dad6d294bbb429f1e66bc25ff97df5f6b20f960e21e68"} Nov 03 08:04:39.969158 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:39.969067 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" podStartSLOduration=1.969054092 podStartE2EDuration="1.969054092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:39.966744537 +0000 UTC m=+3939.900273193" watchObservedRunningTime="2026-11-03 08:04:39.969054092 +0000 UTC m=+3939.902582752" Nov 03 08:04:40.298209 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:40.298131 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:43.697751 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:43.697715 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:04:44.562798 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:44.562758 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:04:44.814187 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:44.814144 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:04:49.297373 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:49.297325 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:49.313869 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:49.313842 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:04:49.934756 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:04:49.934729 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:05:09.139648 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:05:09.139607 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:05:11.498770 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:11.498733 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:05:13.606866 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:13.606809 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:05:22.921107 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:22.920985 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:05:27.083358 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:27.083320 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:05:39.137173 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:05:39.137130 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:05:41.601440 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:41.601357 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss"] Nov 03 08:05:41.612549 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:41.612514 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:05:45.980140 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:45.980105 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:05:56.621599 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.621548 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" podUID="605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" containerName="main" containerID="cri-o://ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8" gracePeriod=45 Nov 03 08:05:56.868599 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.868572 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:05:56.891667 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.891604 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-model-cache\") pod \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " Nov 03 08:05:56.891782 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.891677 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tls-certs\") pod \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " Nov 03 08:05:56.891782 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.891730 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tmp-dir\") pod \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " Nov 03 08:05:56.891782 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.891754 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-kube-api-access-h4ntn\" (UniqueName: \"kubernetes.io/projected/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-kube-api-access-h4ntn\") pod \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " Nov 03 08:05:56.891931 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.891785 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-home\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-home\") pod \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " Nov 03 08:05:56.891931 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.891839 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-dshm\") pod \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\" (UID: \"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9\") " Nov 03 08:05:56.891931 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.891910 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-model-cache" pod "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" (UID: "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:05:56.892128 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.892030 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tmp-dir" pod "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" (UID: "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:05:56.892186 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.892129 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-home" pod "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" (UID: "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:05:56.892186 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.892166 4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-model-cache\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:05:56.892186 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.892178 4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tmp-dir\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:05:56.894119 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.894092 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tls-certs" pod "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" (UID: "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:05:56.894222 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.894115 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-dshm" pod "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" (UID: "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:05:56.894222 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.894138 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-kube-api-access-h4ntn" pod "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" (UID: "605349c2-1d64-43f0-afb2-9b2ad6e5a7c9"). InnerVolumeSpecName "kube-api-access-h4ntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:05:56.993031 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.992910 4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-dshm\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:05:56.993031 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.992949 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-tls-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:05:56.993031 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.992966 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4ntn\" (UniqueName: \"kubernetes.io/projected/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-kube-api-access-h4ntn\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:05:56.993031 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:56.992980 4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9-home\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:05:57.360791 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.360745 4925 generic.go:356] "Generic (PLEG): container finished" podID="605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" containerID="ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8" exitCode=0 Nov 03 08:05:57.361025 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.360795 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" Nov 03 08:05:57.361025 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.360831 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" event={"ID":"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9","Type":"ContainerDied","Data":"ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8"} Nov 03 08:05:57.361025 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.360885 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss" event={"ID":"605349c2-1d64-43f0-afb2-9b2ad6e5a7c9","Type":"ContainerDied","Data":"9eebfa9157f101126d4dad6d294bbb429f1e66bc25ff97df5f6b20f960e21e68"} Nov 03 08:05:57.361025 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.360904 4925 scope.go:122] "RemoveContainer" containerID="ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8" Nov 03 08:05:57.373825 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.373807 4925 scope.go:122] "RemoveContainer" containerID="ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8" Nov 03 08:05:57.374119 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:05:57.374099 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8\": container with ID starting with ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8 not found: ID does not exist" containerID="ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8" Nov 03 08:05:57.374185 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.374130 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8"} err="failed to get container status \"ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8\": rpc error: code = NotFound desc = could not find container \"ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8\": container with ID starting with ae09e4940374476f01d7fbc71ac9b4f6a1462c99e7962320bb213089a9602da8 not found: ID does not exist" Nov 03 08:05:57.531348 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.531315 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss"] Nov 03 08:05:57.537078 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:57.537053 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0730fa14-kserve-c8b597997-2qvss"] Nov 03 08:05:58.190952 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:58.190916 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" path="/var/lib/kubelet/pods/605349c2-1d64-43f0-afb2-9b2ad6e5a7c9/volumes" Nov 03 08:05:58.586758 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:05:58.586723 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:06:09.138023 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:06:09.137963 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:06:09.658500 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:09.658465 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:06:13.875766 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:13.875723 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:31.497244 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.497211 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:06:31.760254 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.760160 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq"] Nov 03 08:06:31.760603 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.760587 4925 cpu_manager.go:414] "containerMap: removing container" podUID="605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" containerName="main" Nov 03 08:06:31.760603 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.760603 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" containerName="main" Nov 03 08:06:31.760755 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.760681 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="605349c2-1d64-43f0-afb2-9b2ad6e5a7c9" containerName="main" Nov 03 08:06:31.763936 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.763915 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.846754 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.846716 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-maas-api-tls\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.846944 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.846771 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-metrics-certs\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.847038 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.846934 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9652faf-f280-4b39-9de7-702fa2f7829a-openshift-service-ca\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.847038 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.846995 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n292g\" (UniqueName: \"kubernetes.io/projected/e9652faf-f280-4b39-9de7-702fa2f7829a-kube-api-access-n292g\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.871746 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.871709 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 08:06:31.872253 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.871868 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 08:06:31.872524 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.872187 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6zz26\"" Nov 03 08:06:31.875376 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.872235 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 08:06:31.875632 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.872277 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 08:06:31.877430 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.877408 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq"] Nov 03 08:06:31.948137 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.948099 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n292g\" (UniqueName: \"kubernetes.io/projected/e9652faf-f280-4b39-9de7-702fa2f7829a-kube-api-access-n292g\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.948340 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.948170 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-maas-api-tls\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.948340 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.948236 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-metrics-certs\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.948460 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.948344 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9652faf-f280-4b39-9de7-702fa2f7829a-openshift-service-ca\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.949381 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.949344 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9652faf-f280-4b39-9de7-702fa2f7829a-openshift-service-ca\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.951122 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.951093 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-metrics-certs\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:31.951237 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:31.951179 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-maas-api-tls\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:32.011963 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:32.011892 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n292g\" (UniqueName: \"kubernetes.io/projected/e9652faf-f280-4b39-9de7-702fa2f7829a-kube-api-access-n292g\") pod \"maas-api-78f9b5fd67-8n4wq\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:32.077572 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:32.077546 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:32.452364 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:32.452334 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq"] Nov 03 08:06:32.453253 ip-10-0-136-184 kubenswrapper[4925]: W1103 08:06:32.453220 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9652faf_f280_4b39_9de7_702fa2f7829a.slice/crio-71e30d13b597d465ea24335201d7321f357733aae168b75e2ebad24fc83bc1f0 WatchSource:0}: Error finding container 71e30d13b597d465ea24335201d7321f357733aae168b75e2ebad24fc83bc1f0: Status 404 returned error can't find the container with id 71e30d13b597d465ea24335201d7321f357733aae168b75e2ebad24fc83bc1f0 Nov 03 08:06:32.454407 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:32.454387 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:06:32.562371 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:32.562335 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" event={"ID":"e9652faf-f280-4b39-9de7-702fa2f7829a","Type":"ContainerStarted","Data":"71e30d13b597d465ea24335201d7321f357733aae168b75e2ebad24fc83bc1f0"} Nov 03 08:06:33.568730 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:33.568696 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" event={"ID":"e9652faf-f280-4b39-9de7-702fa2f7829a","Type":"ContainerStarted","Data":"852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522"} Nov 03 08:06:33.569159 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:33.568827 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:33.690976 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:33.690919 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" podStartSLOduration=1.86634924 podStartE2EDuration="2.690905396s" totalImagesPullingTime="824.556156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:33.690149934 +0000 UTC m=+4053.623678603" watchObservedRunningTime="2026-11-03 08:06:33.690905396 +0000 UTC m=+4053.624434051" Nov 03 08:06:35.996596 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:35.996555 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:38.357749 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:38.357706 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:38.968070 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:38.968000 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:39.140889 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:06:39.140848 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:06:39.180742 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:39.180712 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:39.578851 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:39.578822 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:40.492483 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.492438 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn"] Nov 03 08:06:40.496164 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.496136 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.603263 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.603234 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn"] Nov 03 08:06:40.627888 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.627857 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b86ead5-05a5-4fd0-9fce-54e4c4581735-maas-api-tls\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.628076 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.628030 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wcj\" (UniqueName: \"kubernetes.io/projected/1b86ead5-05a5-4fd0-9fce-54e4c4581735-kube-api-access-k5wcj\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.628076 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.628066 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b86ead5-05a5-4fd0-9fce-54e4c4581735-metrics-certs\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.628176 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.628132 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b86ead5-05a5-4fd0-9fce-54e4c4581735-openshift-service-ca\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.728899 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.728865 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b86ead5-05a5-4fd0-9fce-54e4c4581735-maas-api-tls\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.729085 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.728935 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wcj\" (UniqueName: \"kubernetes.io/projected/1b86ead5-05a5-4fd0-9fce-54e4c4581735-kube-api-access-k5wcj\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.729085 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.728951 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b86ead5-05a5-4fd0-9fce-54e4c4581735-metrics-certs\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.729085 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.728979 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b86ead5-05a5-4fd0-9fce-54e4c4581735-openshift-service-ca\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.729625 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.729598 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b86ead5-05a5-4fd0-9fce-54e4c4581735-openshift-service-ca\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.731437 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.731410 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b86ead5-05a5-4fd0-9fce-54e4c4581735-maas-api-tls\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.731540 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.731436 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b86ead5-05a5-4fd0-9fce-54e4c4581735-metrics-certs\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.794107 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.794040 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wcj\" (UniqueName: \"kubernetes.io/projected/1b86ead5-05a5-4fd0-9fce-54e4c4581735-kube-api-access-k5wcj\") pod \"maas-api-5bcf779ff9-8wkcn\" (UID: \"1b86ead5-05a5-4fd0-9fce-54e4c4581735\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:40.809710 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:40.809689 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:41.185564 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:41.185538 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn"] Nov 03 08:06:41.186770 ip-10-0-136-184 kubenswrapper[4925]: W1103 08:06:41.186747 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b86ead5_05a5_4fd0_9fce_54e4c4581735.slice/crio-00c18dc32d12dad8bd75bf92e144facdacd4a81fee01195f4eb9e38ef03540fc WatchSource:0}: Error finding container 00c18dc32d12dad8bd75bf92e144facdacd4a81fee01195f4eb9e38ef03540fc: Status 404 returned error can't find the container with id 00c18dc32d12dad8bd75bf92e144facdacd4a81fee01195f4eb9e38ef03540fc Nov 03 08:06:41.612240 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:41.612200 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" event={"ID":"1b86ead5-05a5-4fd0-9fce-54e4c4581735","Type":"ContainerStarted","Data":"00c18dc32d12dad8bd75bf92e144facdacd4a81fee01195f4eb9e38ef03540fc"} Nov 03 08:06:42.362507 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:42.362467 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:42.600698 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:42.600652 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:42.618701 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:42.618618 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" event={"ID":"1b86ead5-05a5-4fd0-9fce-54e4c4581735","Type":"ContainerStarted","Data":"a4ed862ef9721fa342267b771f2acf1ac5ba5a01c6f014b264cadd72ea89cb1b"} Nov 03 08:06:42.618701 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:42.618694 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:42.750545 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:42.750496 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" podStartSLOduration=1.936639332 podStartE2EDuration="2.750483718s" totalImagesPullingTime="813.844386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:42.748524513 +0000 UTC m=+4062.682053169" watchObservedRunningTime="2026-11-03 08:06:42.750483718 +0000 UTC m=+4062.684012406" Nov 03 08:06:45.651909 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:45.651867 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:45.890320 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:45.890278 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:48.046872 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:48.046837 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:48.229346 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:48.229312 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:48.414616 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:48.414579 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:48.629411 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:48.629380 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-8wkcn" Nov 03 08:06:48.804413 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:48.804329 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq"] Nov 03 08:06:48.804593 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:48.804524 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" podUID="e9652faf-f280-4b39-9de7-702fa2f7829a" containerName="maas-api" containerID="cri-o://852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522" gracePeriod=30 Nov 03 08:06:49.065614 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.065593 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:49.107688 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.107661 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9652faf-f280-4b39-9de7-702fa2f7829a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9652faf-f280-4b39-9de7-702fa2f7829a-openshift-service-ca\") pod \"e9652faf-f280-4b39-9de7-702fa2f7829a\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " Nov 03 08:06:49.107806 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.107745 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-metrics-certs\") pod \"e9652faf-f280-4b39-9de7-702fa2f7829a\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " Nov 03 08:06:49.107806 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.107765 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9652faf-f280-4b39-9de7-702fa2f7829a-kube-api-access-n292g\" (UniqueName: \"kubernetes.io/projected/e9652faf-f280-4b39-9de7-702fa2f7829a-kube-api-access-n292g\") pod \"e9652faf-f280-4b39-9de7-702fa2f7829a\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " Nov 03 08:06:49.107885 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.107824 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-maas-api-tls\") pod \"e9652faf-f280-4b39-9de7-702fa2f7829a\" (UID: \"e9652faf-f280-4b39-9de7-702fa2f7829a\") " Nov 03 08:06:49.108076 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.108051 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9652faf-f280-4b39-9de7-702fa2f7829a-openshift-service-ca" pod "e9652faf-f280-4b39-9de7-702fa2f7829a" (UID: "e9652faf-f280-4b39-9de7-702fa2f7829a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:06:49.110059 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.109996 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-maas-api-tls" pod "e9652faf-f280-4b39-9de7-702fa2f7829a" (UID: "e9652faf-f280-4b39-9de7-702fa2f7829a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:06:49.110059 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.110042 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9652faf-f280-4b39-9de7-702fa2f7829a-kube-api-access-n292g" pod "e9652faf-f280-4b39-9de7-702fa2f7829a" (UID: "e9652faf-f280-4b39-9de7-702fa2f7829a"). InnerVolumeSpecName "kube-api-access-n292g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:06:49.110244 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.110075 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-metrics-certs" pod "e9652faf-f280-4b39-9de7-702fa2f7829a" (UID: "e9652faf-f280-4b39-9de7-702fa2f7829a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:06:49.209401 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.209372 4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-metrics-certs\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:49.209401 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.209399 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n292g\" (UniqueName: \"kubernetes.io/projected/e9652faf-f280-4b39-9de7-702fa2f7829a-kube-api-access-n292g\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:49.209401 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.209409 4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9652faf-f280-4b39-9de7-702fa2f7829a-maas-api-tls\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:49.209702 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.209418 4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9652faf-f280-4b39-9de7-702fa2f7829a-openshift-service-ca\") on node \"ip-10-0-136-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:49.656251 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.656213 4925 generic.go:356] "Generic (PLEG): container finished" podID="e9652faf-f280-4b39-9de7-702fa2f7829a" containerID="852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522" exitCode=0 Nov 03 08:06:49.656455 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.656263 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" Nov 03 08:06:49.656455 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.656290 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" event={"ID":"e9652faf-f280-4b39-9de7-702fa2f7829a","Type":"ContainerDied","Data":"852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522"} Nov 03 08:06:49.656455 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.656342 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq" event={"ID":"e9652faf-f280-4b39-9de7-702fa2f7829a","Type":"ContainerDied","Data":"71e30d13b597d465ea24335201d7321f357733aae168b75e2ebad24fc83bc1f0"} Nov 03 08:06:49.656455 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.656370 4925 scope.go:122] "RemoveContainer" containerID="852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522" Nov 03 08:06:49.666710 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.666689 4925 scope.go:122] "RemoveContainer" containerID="852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522" Nov 03 08:06:49.667037 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:06:49.666995 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522\": container with ID starting with 852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522 not found: ID does not exist" containerID="852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522" Nov 03 08:06:49.667103 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.667046 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522"} err="failed to get container status \"852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522\": rpc error: code = NotFound desc = could not find container \"852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522\": container with ID starting with 852796c7919df1a0774098a5bf87474e45a059e3ef8d29c5cff4c9760765c522 not found: ID does not exist" Nov 03 08:06:49.826998 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.826968 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq"] Nov 03 08:06:49.878212 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:49.878177 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8n4wq"] Nov 03 08:06:50.190351 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:50.190319 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9652faf-f280-4b39-9de7-702fa2f7829a" path="/var/lib/kubelet/pods/e9652faf-f280-4b39-9de7-702fa2f7829a/volumes" Nov 03 08:06:53.844719 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:53.844675 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:06:56.470354 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:56.470314 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:06:58.961141 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:06:58.961103 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:04.055637 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:04.055599 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:09.137334 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:07:09.137296 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:07:09.636464 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:09.636430 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:14.220919 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:14.220825 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:14.407450 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:14.407412 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:14.871946 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:14.871912 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:20.032890 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:20.032851 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:25.152986 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:25.152938 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:29.662193 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:29.662156 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:30.203645 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:30.203608 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:30.389725 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:30.389688 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:34.301264 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:34.301229 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:07:34.400510 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:34.400469 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:34.691209 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:34.691172 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:39.138338 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:07:39.138295 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:07:39.843665 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:39.843630 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:41.114687 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:41.114649 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:41.752206 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:41.752171 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:41.938519 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:41.938476 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:45.679471 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:45.679416 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:45.891796 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:45.891755 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:48.985780 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:48.985743 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:49.216440 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:49.216385 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:52.087124 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:52.087087 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:52.272634 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:52.272602 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:52.459243 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:52.459153 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:07:57.349267 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:07:57.349229 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:00.211315 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:00.211264 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:08:02.585233 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:02.585195 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:07.781704 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:07.781666 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:09.136476 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:08:09.136431 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:08:10.462481 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:10.462444 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:13.162143 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:13.162091 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:14.561993 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:14.561954 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:18.270322 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:18.270277 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:23.358112 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:23.358077 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:28.468152 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:28.468110 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:33.559964 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:33.559925 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:36.519086 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:36.519042 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:38.662748 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:38.662705 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:39.137724 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:08:39.137689 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:08:43.884893 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:43.884809 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:48.962234 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:48.962178 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:54.155076 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:54.155031 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:08:59.264325 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:08:59.264277 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:02.035750 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:02.035718 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 08:09:02.039216 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:02.039193 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 08:09:02.048459 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:02.048439 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ldktd_452ab275-c3b0-4c31-8603-7af7cd666346/console-operator/2.log" Nov 03 08:09:02.051850 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:02.051828 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pb8jn_e6eb01e1-380a-468c-8097-d34540808454/ovn-acl-logging/0.log" Nov 03 08:09:04.447976 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:04.447922 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:09.137764 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:09:09.137732 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:09:09.568737 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:09.568684 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:14.402212 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:14.402168 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:14.677327 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:14.677245 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:14.912687 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:14.912656 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:19.861119 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:19.861072 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:25.062245 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:25.062202 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:30.159956 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:30.159899 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:30.530589 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:30.530475 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:30.770459 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:30.770427 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:35.635704 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:35.635666 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:39.137455 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:09:39.137412 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:09:40.857597 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:40.857549 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:43.147943 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:43.146676 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:43.343761 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:43.343725 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:50.425501 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:50.425462 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:51.994813 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:51.994777 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:52.284001 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:52.283913 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:53.051200 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:53.051164 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:57.168485 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:57.168455 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:09:59.457070 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:59.457032 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:59.649683 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:59.649628 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:09:59.841584 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:09:59.841542 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:10:09.137424 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:10:09.137386 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:10:09.305509 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:09.305476 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:10:15.304181 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:15.304076 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:10:15.486792 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:15.486737 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:10:22.205863 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:22.205826 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:10:24.622383 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:24.622123 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:10:24.622383 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:24.622202 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz" podUID="03b23567-bced-4d00-977f-73e8ccedce7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:24.670034 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:24.669116 4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:10:24.670034 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:24.669201 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52" podUID="2ff1a6c9-e407-4541-a669-ad0f433d25b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:35.652538 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:35.652484 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:10:39.137173 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:10:39.137139 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:10:43.293810 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:10:43.293778 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:11:09.138376 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:11:09.138335 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:11:15.552738 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:15.552693 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:11:16.001096 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:16.001060 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lwnr4"] Nov 03 08:11:27.557286 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:27.557243 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9shxn_4da3a4da-2a87-4311-886a-7237c8e38e9e/manager/0.log" Nov 03 08:11:29.549154 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:29.549116 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bcf779ff9-8wkcn_1b86ead5-05a5-4fd0-9fce-54e4c4581735/maas-api/0.log" Nov 03 08:11:32.758750 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:32.758716 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-884dw_a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd/manager/0.log" Nov 03 08:11:33.000937 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:33.000903 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wwj4m_2e8cbf75-be5c-471c-96ee-22721f1b276c/manager/0.log" Nov 03 08:11:33.217110 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:33.217083 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-chsvv_77206c00-a1b5-48e9-a136-c87966d96a42/kuadrant-console-plugin/0.log" Nov 03 08:11:33.482585 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:33.482506 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fm5w4_404a8fbd-a76d-4d18-b82e-23e1ace2d745/manager/0.log" Nov 03 08:11:33.796394 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:33.796313 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lwnr4_91b42406-d50b-429d-b9e9-a1079b1dc162/limitador/0.log" Nov 03 08:11:34.042733 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:34.042701 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nsnp2_22b735f0-6048-48a9-aebf-bf4f7270ab02/manager/0.log" Nov 03 08:11:34.872487 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:34.872453 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9_8305581d-8cb9-4e1f-b5b5-c84577035c7d/istio-proxy/0.log" Nov 03 08:11:35.816441 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:35.816412 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-4jgbz_03b23567-bced-4d00-977f-73e8ccedce7c/istio-proxy/0.log" Nov 03 08:11:37.462921 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:37.462887 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-rdzr6_aa09988a-1035-414f-8838-21f9395f72ee/istio-proxy/0.log" Nov 03 08:11:37.927480 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:37.927446 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-xqv52_2ff1a6c9-e407-4541-a669-ad0f433d25b0/istio-proxy/0.log" Nov 03 08:11:38.171417 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:38.171380 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp_a0933a67-48ab-4a25-9e4b-c02b650eefcf/istio-proxy/0.log" Nov 03 08:11:38.873626 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:38.873595 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-699757bbb7-drfpj_84d699e8-7ccc-4e2c-b6c6-e309946bee2e/router/0.log" Nov 03 08:11:39.137624 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:11:39.137547 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:11:53.617180 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:53.617089 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9shxn_4da3a4da-2a87-4311-886a-7237c8e38e9e/manager/0.log" Nov 03 08:11:55.291137 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:55.291096 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bcf779ff9-8wkcn_1b86ead5-05a5-4fd0-9fce-54e4c4581735/maas-api/0.log" Nov 03 08:11:58.289785 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:58.289748 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-884dw_a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd/manager/0.log" Nov 03 08:11:58.549924 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:58.549839 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wwj4m_2e8cbf75-be5c-471c-96ee-22721f1b276c/manager/0.log" Nov 03 08:11:58.780662 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:58.780620 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-chsvv_77206c00-a1b5-48e9-a136-c87966d96a42/kuadrant-console-plugin/0.log" Nov 03 08:11:59.039723 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:59.039688 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fm5w4_404a8fbd-a76d-4d18-b82e-23e1ace2d745/manager/0.log" Nov 03 08:11:59.309068 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:59.308969 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lwnr4_91b42406-d50b-429d-b9e9-a1079b1dc162/limitador/0.log" Nov 03 08:11:59.530069 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:11:59.530025 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nsnp2_22b735f0-6048-48a9-aebf-bf4f7270ab02/manager/0.log" Nov 03 08:12:00.139404 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:00.139369 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9_8305581d-8cb9-4e1f-b5b5-c84577035c7d/istio-proxy/0.log" Nov 03 08:12:01.071757 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:01.071712 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-4jgbz_03b23567-bced-4d00-977f-73e8ccedce7c/istio-proxy/0.log" Nov 03 08:12:02.698723 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:02.698685 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-rdzr6_aa09988a-1035-414f-8838-21f9395f72ee/istio-proxy/0.log" Nov 03 08:12:03.169116 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:03.169081 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-xqv52_2ff1a6c9-e407-4541-a669-ad0f433d25b0/istio-proxy/0.log" Nov 03 08:12:03.407186 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:03.407148 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp_a0933a67-48ab-4a25-9e4b-c02b650eefcf/istio-proxy/0.log" Nov 03 08:12:04.123511 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:04.123474 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-699757bbb7-drfpj_84d699e8-7ccc-4e2c-b6c6-e309946bee2e/router/0.log" Nov 03 08:12:09.137441 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:12:09.137401 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:12:15.952134 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:15.952094 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:12:39.137696 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:12:39.137636 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:12:42.359613 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:12:42.359576 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:13:09.138467 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:13:09.138432 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:13:16.289394 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:16.289299 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:13:16.637905 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:16.637868 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:13:18.192580 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:18.192547 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:13:18.387077 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:18.387040 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:13:27.094543 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:27.094508 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dtzxb_ca92ce4d-7cf4-4da5-8609-920db260c2d3/global-pull-secret-syncer/0.log" Nov 03 08:13:28.930574 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:28.930547 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4jk57_4e7d1d1e-2fb6-412d-8597-f6f4882d0389/connection-checker/0.log" Nov 03 08:13:29.354755 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:29.354728 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5ndhb_25276ee9-63e0-4d32-834e-8223c741b724/connection-checker/0.log" Nov 03 08:13:29.781123 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:29.781049 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hzqt4_40c1e3ef-149c-401f-8e88-8bdafa0faae6/connection-checker/0.log" Nov 03 08:13:30.254510 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:30.254476 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" Nov 03 08:13:31.936967 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:31.936930 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-184.us-west-1.compute.internal_1f565955bc42ade3fa3b9ef6d6540a53/haproxy/0.log" Nov 03 08:13:36.770354 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:36.770323 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-884dw_a5c4f39f-4f97-4dfa-9aaa-d24f861a97cd/manager/0.log" Nov 03 08:13:37.199832 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.199783 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wwj4m_2e8cbf75-be5c-471c-96ee-22721f1b276c/manager/0.log" Nov 03 08:13:37.378762 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.378727 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct"] Nov 03 08:13:37.379182 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.379167 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e9652faf-f280-4b39-9de7-702fa2f7829a" containerName="maas-api" Nov 03 08:13:37.379247 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.379183 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9652faf-f280-4b39-9de7-702fa2f7829a" containerName="maas-api" Nov 03 08:13:37.379285 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.379271 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9652faf-f280-4b39-9de7-702fa2f7829a" containerName="maas-api" Nov 03 08:13:37.382329 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.382313 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.437286 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.437249 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lpwd\"/\"kube-root-ca.crt\"" Nov 03 08:13:37.437424 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.437311 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lpwd\"/\"openshift-service-ca.crt\"" Nov 03 08:13:37.437424 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.437345 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9lpwd\"/\"default-dockercfg-blmhb\"" Nov 03 08:13:37.492062 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.491970 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct"] Nov 03 08:13:37.532377 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.532348 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-sys\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.532520 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.532397 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-podres\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.532520 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.532427 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-proc\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.532520 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.532509 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-lib-modules\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.532634 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.532551 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br9xg\" (UniqueName: \"kubernetes.io/projected/6f263824-f6b5-4f92-aba7-af6b2947938a-kube-api-access-br9xg\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.629904 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.629875 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-chsvv_77206c00-a1b5-48e9-a136-c87966d96a42/kuadrant-console-plugin/0.log" Nov 03 08:13:37.633275 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633255 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9xg\" (UniqueName: \"kubernetes.io/projected/6f263824-f6b5-4f92-aba7-af6b2947938a-kube-api-access-br9xg\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633367 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633297 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-sys\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633454 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633433 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-podres\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633518 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633469 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-proc\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633518 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633477 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-podres\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633518 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633440 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-sys\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633662 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633535 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-proc\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633662 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633573 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-lib-modules\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.633736 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.633671 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f263824-f6b5-4f92-aba7-af6b2947938a-lib-modules\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.697122 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.697095 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9xg\" (UniqueName: \"kubernetes.io/projected/6f263824-f6b5-4f92-aba7-af6b2947938a-kube-api-access-br9xg\") pod \"perf-node-gather-daemonset-4wkct\" (UID: \"6f263824-f6b5-4f92-aba7-af6b2947938a\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:37.994852 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:37.994822 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:38.101069 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.101019 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fm5w4_404a8fbd-a76d-4d18-b82e-23e1ace2d745/manager/0.log" Nov 03 08:13:38.371226 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.371205 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct"] Nov 03 08:13:38.372202 ip-10-0-136-184 kubenswrapper[4925]: W1103 08:13:38.372176 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f263824_f6b5_4f92_aba7_af6b2947938a.slice/crio-c6435006936400d91702093b284d93fcf422b635c28bc94ed08ce2e47134bab8 WatchSource:0}: Error finding container c6435006936400d91702093b284d93fcf422b635c28bc94ed08ce2e47134bab8: Status 404 returned error can't find the container with id c6435006936400d91702093b284d93fcf422b635c28bc94ed08ce2e47134bab8 Nov 03 08:13:38.478237 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.478210 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lwnr4_91b42406-d50b-429d-b9e9-a1079b1dc162/limitador/0.log" Nov 03 08:13:38.835946 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.835917 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" event={"ID":"6f263824-f6b5-4f92-aba7-af6b2947938a","Type":"ContainerStarted","Data":"8c7ef55290f1639f1044f14d8b37ce326586c95f0969dac7c614cca3a37d1a27"} Nov 03 08:13:38.836147 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.835966 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" event={"ID":"6f263824-f6b5-4f92-aba7-af6b2947938a","Type":"ContainerStarted","Data":"c6435006936400d91702093b284d93fcf422b635c28bc94ed08ce2e47134bab8"} Nov 03 08:13:38.836147 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.835996 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:38.957753 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.957682 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" podStartSLOduration=1.957667722 podStartE2EDuration="1.957667722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:38.955414041 +0000 UTC m=+4478.888942698" watchObservedRunningTime="2026-11-03 08:13:38.957667722 +0000 UTC m=+4478.891196377" Nov 03 08:13:38.976448 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:38.976421 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nsnp2_22b735f0-6048-48a9-aebf-bf4f7270ab02/manager/0.log" Nov 03 08:13:39.137591 ip-10-0-136-184 kubenswrapper[4925]: E1103 08:13:39.137504 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jjq57" containerName="extractor" Nov 03 08:13:44.569503 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:44.569468 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-n6xht_902e5dbc-172c-496e-afaf-2e847ba1f852/cluster-monitoring-operator/0.log" Nov 03 08:13:44.852974 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:44.852891 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-4wkct" Nov 03 08:13:47.069422 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:47.069393 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gqd4_99c81080-acdb-4468-8ef4-a2e35d7b2969/node-exporter/0.log" Nov 03 08:13:47.486788 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:47.486756 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gqd4_99c81080-acdb-4468-8ef4-a2e35d7b2969/kube-rbac-proxy/0.log" Nov 03 08:13:47.911733 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:47.911706 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7gqd4_99c81080-acdb-4468-8ef4-a2e35d7b2969/init-textfile/0.log" Nov 03 08:13:48.379070 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:48.379042 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:13:48.562262 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:13:48.562231 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gwnd_f7781680-4a88-4a3f-8fc1-61c702b89f63/konnectivity-agent/0.log" limit=1024 Nov 03 08:14:00.208031 ip-10-0-136-184 kubenswrapper[4925]: I1103 08:14:00.207974 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9_8305581d-8cb9-4e1f-b5b5-c84577035c7d/istio-proxy/0.log"